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Executive  Summary 


Augmented  reality  (AR)  systems  have  been  used  for  years,  but  due  to  their  cost  and 
complexity,  most  AR  systems  were  developed  for  very  specific  applications.  Soon,  low-cost, 
adaptive  AR  systems  will  enter  the  consumer  market  place.  As  commercially  viable  AR  systems 
become  available,  many  market  analysts  predict  that  AR  systems  will  become  regular  features  of 
most  people’s  daily  lives  and  work  environments.  Just  as  the  regular  use  of  personal  computers 
and  smartphones  altered  human  behaviors  the  habitual  use  of  AR  systems  will  also  transform 
how  people  interact  with  their  environments  and  each  other. 

Augmented  reality  systems  merge  virtual  content  with  a  user’s  real-world  perceptual 
experiences.  Information  from  one’s  surroundings  is  captured  by  AR  systems  through  a 
multitude  of  sensors  incorporated  into  AR  devices.  Various  AR  applications  then  process  this 
captured  data  so  as  to  present  AR  users  with  enhancing  information  in  real-time.  This  enhancing 
information  is  typically  presented  by  means  of  an  optical  display  device  worn  as  glasses, 
goggles,  or  a  visor.  Depending  on  the  type  of  AR  display  device  this  information  can  be 
presented  in  either  two  or  three  dimensions  and  can  either  float  over  a  user’s  perceptual  field-of- 
view  or  be  digitally  anchored  so  as  to  appear  like  a  real  object  for  the  AR  user  to  interact  with. 

Continuous  access  to  AR  content  has  the  potential  to  affect  how  AR  users  think,  learn, 
and  act.  Augmented  reality  systems  impact  what  captures  users’  attention  and  they  alter  how  AR 
users  direct  their  attention.  Furthermore,  the  more  AR  users  interact  with  AR  systems,  the  more 
they  will  adjust  their  information  behaviors.  They  will  develop  new  learning  styles  to  leverage 
the  capabilities  of  these  systems.  Likewise,  through  programmed  feedback  mechanisms,  AR 
systems  themselves  will  learn  and  adapt  to  their  users’  idiosyncratic  information  behaviors. 
Moreover,  the  habitual  use  of  AR  systems  will  likely  impact  AR  users’  decision-making  because 
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it  will  offer  instantaneous  access  to  expertise  and  an  expanded  range  of  knowledge  from  which 
to  reference.  The  use  of  AR  systems  will  change  how  AR  users  encode,  interpret,  and  interact 
with  their  world. 

Given  that  AR  systems  appear  to  be  on  the  cusp  of  large-scale  commercial  adoption  it  is 
important  that  the  military  consider  the  likely  ramifications  this  technology  will  have.  The 
following  paper  argues  that  AR  systems  will  serve  as  the  next  revolution  in  information 
technologies  and  as  such  it  should  be  leveraged  by  the  U.S.  military  to  maintain  its  relative 
information  superiority  advantages.  The  use  of  AR  systems  at  the  tactical  level  and  operational 
level  would  provide  each  branch  of  service  marked  improvements  in  a  variety  of  tasks. 
Furthermore,  the  use  of  AR  systems  by  operational  leaders  and  their  staffs  would  provide 
expanded  abilities  to  understand  complex  operating  environments  and  new  means  to  command 
and  control  tactical  forces. 

If  the  military  determines  to  integrate  AR  systems  into  its  daily  garrison  activities,  and 
onto  its  combat  battlefields,  it  will  have  to  overcome  a  number  of  challenges  and  mitigate  certain 
risks.  The  greatest  limiting  factor  to  introducing  such  a  disruptive  technology  as  AR  will  be 
overcoming  policy  restrictions  particularly  as  they  relate  to  privacy,  security,  and  data- 
classification.  However,  other  difficulties  will  exist  from  overcoming  anti-technology  biases  to 
allaying  fears  of  increased  micromanagement  due  to  pervasive  presence  of  communication 
systems.  In  the  end,  however,  each  of  these  challenges  comes  with  an  opportunity  for  increased 
combat  effectiveness. 

The  time  has  come  for  the  military  to  begin  planning  for  a  future  with  AR  systems.  It  can 
best  prepare  by  anticipating  future  back-end  infrastructure  requirements  and  developing  battle 
management  systems  that  readily  share  information  due  to  prescribed  interoperability  and 
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standardization  requirements.  Likewise,  the  military  should  refine  its  overall  information 
management  plan  to  ensure  AR  systems  can  maximize  their  ability  to  access  and  contribute  to  it. 
Furthermore,  the  military  should  define  its  unique  requirements  for  AR  systems  now  to  ensure 
they  have  access  to  future  commercial  offerings  that  are  both  intuitively  controlled  and  designed 
for  tactical  use.  Finally,  the  military  must  consider  developing  new  organizational  structures, 
policies,  and  doctrine  to  prepare  for  the  inevitable  changes  AR  systems  will  inspire. 


Introduction 


U.S.  military  supremacy  depends  on  the  relative  technological  superiority  of  its  combat 
systems.  As  technologies  advance,  the  U.S.  seeks  to  retain  its  military  advantage  by  integrating 
new  inventions  that  will  make  its  forces  more  lethal  and  better  able  to  survive  in  combat.  It 
makes  these  investments  so  as  to  maintain  its  relative  advantage  in  combat  effectiveness  over 
other  militaries.  Even  though  basic  materials  have  improved  and  mechanical  engineering  has 
modernized  over  the  past  fifty-years,  the  one  particular  technology  that  has  transformed  the  U.S. 
military  more  than  any  other  is  information  technology  (IT). 

Since  the  advent  of  the  micro-chip,  the  rate  of  improvement  in  IT  has  far  surpassed 
advances  in  other  technologies.  IT  advances  at  an  exponential  rate.1 2  Gordon  Moore  noted  early 
in  the  start  of  the  micro-chip  revolution,  IT  components  appeared  to  double  their  capacity  nearly 
every  12-18  months  (i.e.  processing  speed,  memory,  etc.).  At  the  same  time,  the  cost  of  these 
components  decreases  with  time.  Recognizing  this,  the  U.S.  military  has  viewed  advancements 
in  IT  as  a  driving  force  for  change  and  improvement  in  its  combat  systems.  From  early 
investments  in  battlefield  management  systems  to  the  development  of  modern  precision-guided 
munitions,  the  U.S.  military  has  sought  to  leverage  emerging  IT  systems  and  technologies  to 
improve  its  forces  across  all  echelons. 

The  central  way  that  IT  systems  increase  the  combat  effectiveness  of  military  forces  is  by 
providing  for  and  applying  information  superiority.  Information  superiority  is  the  ability  to 
collect,  process,  and  share  information  more  effectively  than  an  adversary.'  The  U.S.  military  has 
determined  that  it  is  absolutely  essential  to  maintain  information  superiority  over  adversaries. 

1  Ray  Kurzweill,  "The  Law  of  Accelerating  Returns/'  (Kurzweil  Accelerating  Intelligence,  March  7,  2001, 
http://www.kurzweilai.net/the-law-of-accelerating-returns,  accessed  October  8,  2015. 

2  David  S.  Alberts,  "Key  Concepts  for  Information  Superiority,"  (Military  Technology,  Vol  11.,  2008),  p.  75. 
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The  U.S.  military  regularly  delivers  precision-strike  munitions  with  long  stand-off  ranges, 
employs  network-centric  forces,  and  conducts  effects-based  operations  in  order  to  achieve  U.S. 
objectives.  The  command  and  control  (C2)  challenges  in  coordinating  these  types  of  operations 
require  robust  IT  systems.  As  such,  the  U.S.  military  continues  to  seek  out  new  IT  systems 
capable  of  improving  its  performance. 

One  emerging  IT  tool  that  promises  to  increase  the  combat  effectiveness  of  U.S.  forces  is 
augmented  reality  (AR).  Augmented  reality  systems  are  an  information  technology  used  to 
enhance  real-world  experiences.  Unlike  virtual  reality  systems  that  immerse  users  in  completely 
computer  generated  experience,  AR  provides  virtual  content  over  AR  users’  perceptual  field-of- 
view  (FOV).  They  create  an  interactive  experience  between  AR  users,  their  environments,  and 
virtual  content. 

Leading  commercial  IT  companies  and  the  defense  industry  recognize  the  potential  of 
AR  and  have  invested  heavily  in  developing  commercial  AR  hardware  and  operating  systems. 
Products  such  as  Google  Glass,  Sony’s  SmartEyeglasses,  and  Microsoft’s  HoloLens  have 
emerged  as  commercially  viable  AR  devices.3 4  Other  AR  display  devices  developed  by  Epson, 
Inforod,  Vuzix,  Mirama,  and  MetaPro  are  demonstrating  the  potential  of  augmented  reality.5  In 
the  defense  sector,  BAE  Systems,  Rockwell  Collins,  and  Lockheed  Martin  have  engineered  AR 
capable  devices  mostly  for  pilots  and  vehicle  operators.  As  these  and  other  AR  devices  make 
their  way  into  commercial  markets  AR  software  engineers  see  the  value  in  investing  in  the 
development  of  AR  applications.  They  are  accelerating  their  investment  in  AR  app  development 


3  Ibid. 

4  Jessi  Hempel,  "Project  Hololens:  Our  Exclusive  Hand-on  with  Microsoft's  Holographic  Goggles,"  (Wired,  January 
15.2015). 

5  Anand  Dibble,  12  Promising  Augmented  Reality  Devices  -  The  State  ofAR  Hardware,  (Brainberry  Global, 
September  16,  2014,  http://brainberrvglobal.com/12-promising-augmented-reality-devices-state-ar-hardware/, 
accessed  August  20,  2015). 
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due  to  the  anticipation  of  large-scale  consumer  adoption  of  AR  systems.6  Furthermore,  annual 
revenue  from  the  sale  of  AR  applications  continues  to  grow  and  is  expected  to  increase  from 
$247  million  in  2014  to  over  $2.4  billion  in  20 15. 7  In  fact,  some  analysts  believe  that  virtual  and 
augmented  reality  technologies  will  be  worth  as  much  as  $150  billion  as  early  as  2020. 8 

A  large  reason  the  AR  commercial  sector  has  reached  a  tipping  point  for  large-scale 
consumer  adoption  is  due  to  the  maturation  of  the  component  technologies  incorporated  into  AR 
systems.  The  director  of  Intel  Labs  recently  noted  that  the  computer  technologies, 
communications  technologies,  and  sensor  technologies  have  all  matured  to  a  point  where  the 
potential  for  high-volume  consumer  wearable  AR  devices  is  practicable.9  The  proliferation  of 
digital  sensors  and  the  increase  in  their  capabilities  enabled  AR  systems  with  new  means  to 
encode  the  world.  In  addition,  the  expansion  of  high-speed  mobile  internet  enables  AR  systems 
to  access  external  data  sources  and  computational  resources  it  requires.  In  addition,  the 
emergence  of  super-computational  processing  that  can  render  high-definition  2D  and  3D 
information  in  real-time  enables  the  high-quality  AR  interactive  experience  consumers  have 
come  to  expect. 

The  convergence  of  viable  AR  devices,  a  robust  community  of  AR  software  developers,  a 
growing  demand  signal  from  consumers,  and  the  emergence  of  the  back-end  IT  infrastructure 
AR  requires,  has  convinced  many  commercial  analysts  that  AR  will  soon  enter  a  period  of  rapid 


6  ABI,  Cloud  Strategies  for  Augmented  Reality,  (ABI,  July  3,  2013,  https://www.abiresearch.com/press/developers- 
to-invest-25-billion-in-augmented-reali/,  accessed  August  20,  2015). 

7  Juniper  Research,  Mobile  Augmented  Reality:  Smartphones,  Tablets  and  Smart  Glasses  2013-2018,  (Juniper 
Research,  http://www.iuniperresearch.com/researchstore/enabling-technologies/augmented-reality/consumer- 

enterprise-vehicles,  2013). 

8  Digi-Capital  Report,  “Augmented/Virtual  Reality  Report  Q2”  (Digi-Capital,  April  2015,|http://www.digi- 
capital.eom/news/2015/04/augmentedvirtual-realitv-to-hit-150-billion-disrupting-mobile-bv-2020/#.ViwTSsuFNis, 

accessed  September  12,  2015). 

9  Larry  Greenemeier,  “The  Future  of  Smart  Glasses,”  (Scientific  American,  Vol.  309,  July,  2013). 
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growth.  AR  has  reached  a  tipping  point  and  many  believe  AR  will  prove  to  be  a  “platform 
integral  to  the  consumer’s  ecosystem  and  daily  life”.10 

The  likely  symbiotic  relationship  between  humankind  and  AR  systems  has  the  potential 
to  fundamentally  change  how  people  interact  with  each  other  and  their  environments.  As  such, 
the  time  has  come  for  organizations  to  consider  the  benefits  and  challenges  their  personnel  will 
experience  once  AR  systems  are  fully  adopted.  This  paper  will  explain  these  benefits  and 
challenges  as  they  relate  to  the  use  of  AR  systems  by  the  U.S.  military.  It  will  also  explain  the 
specific  cognitive  factors  affected  by  AR  systems  and  what  this  means  for  human  attention, 
learning,  and  decision-making.  Furthermore,  this  paper  will  explain  how  the  integration  of  AR 
systems  will  improve  decision-making  at  the  tactical  and  operational  levels  -  and  thereby 
increase  the  overall  combat  effectiveness  of  U.S.  forces.  Finally,  it  will  offer  recommendations 
for  how  the  U.S.  military  can  best  prepare  for  a  future  world  with  AR  systems. 


10  Juniper  Research,  2013. 
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Background:  Information  Superiority  and  Combat  Effectiveness 


Those  organizations  that  adapt  and  exploit  advances  in  information  technology  tools  gain 
a  competitive  advantage  over  those  that  fail  to  do  so.11  Information  technology  systems  enable 
organizations  to  achieve  information  superiority  so  as  to  make  more  effective  and  efficient 
decisions.  "  The  U.S.  military  recognizes  this  and  continues  to  develop,  procure,  and  integrate 
systems  it  believes  will  benefit  its  tactical  combat  units  and  operational  leaders.  Ultimately,  as 
the  capabilities  of  information  systems  increase,  they  enable  military  forces  at  both  the 

1  T 

operational  and  tactical  levels  to  better  orient  and  direct  forces  toward  emerging  threats. 

The  U.S.  military  has  a  long  history  of  researching  and  developing  new  IT  systems  to 
help  it  achieve  information  superiority.  As  new  innovations  emerge  it  continues  to  seek  ways  to 
leverage  their  capabilities  so  as  to  maintain  its  information  superiority  advantage.14  It  does  this 
because  military  leaders  need  both  access  to  timely  and  accurate  information  about  their 
operating  environment  and  the  means  to  communicate  their  vision  to  decision-makers  and  their 
forces.15  During  combat,  information  superiority  -  i.e.,  the  ability  to  provide  vital  information  to 
forces  with  “superior  timeliness,  relevance,  accuracy,  and  comprehensiveness”  than  an  adversary 


11  David  S.  Alberts,  and  Richard  E.  Hayes,  Understanding  Command  and  Control  (Washington,  D.C.:  DoD 
Command  and  Control  Research  Program,  2006). 

12  Ibid. 

13  JCS,  JF-2020,  2012. 

14  Walter  Perry,  David  Signori,  and  John  Boon,  Exploring  Information  Superiority:  A  Methodology  for  Measuring 
the  Quality  of  Information  and  Its  Impact  on  Shared  Awareness ,  (Rand  Corporation  report  for  U.S.  SECDEF,  2004). 

15  JCS,  Capstone  Concept  for  Joint  Operations:  Joint  Force  2020,  (Washington,  D.C.:  Joint  Chiefs  of  Staff,  2012). 
See  also  David  S.  Alberts,  The  Agility  Advantage,  (Washington,  D.C.:  DoD  Command  and  Control  Research 
Program,  2011). 
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can  achieve  -  can  be  decisive  in  determining  the  outcome.16  Now,  more  than  ever  information 
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superiority  is  a  prerequisite  for  mission  success. 

At  the  tactical  level,  information  superiority  can  determine  survival  or  peril.  In  combat, 
information  superiority  enables  military  forces  to  locate  and  target  enemy  forces  faster  than  they 
can  do  the  same.  Speed  in  decision-making  allows  one  to  disrupt  an  adversary’s  ability  to 
respond  and  react  and  helps  tactical  units  gain  and  maintain  the  tactical  imperative.  At  the 
tactical  level,  information  superiority  also  helps  combat  forces  understand  the  location  and 
disposition  of  friendly  forces.  This  increases  their  overall  unity-of-effort  and  prevents  inadvertent 
accidents  and  fratricides.  Information  superiority,  in  addition  to  aid  combat  forces  defeat  an 
adversary,  and  support  friendly  forces,  can  also  help  to  prevent  collateral  damage  and  civilian 
casualties.  Beyond  the  immediate  life-and-death  benefits  at  the  tactical  level  of  information 
superiority  -  it  can  increase  combat  effectiveness  at  a  much  broader  scale  operationally. 

At  the  operational  level  of  war,  information  superiority  enables  the  C2  efforts  of  Joint- 
Task-Force  operational  commands.  It  allows  operational  commands  the  ability  quickly  assemble 
a  full  range  of  joint  weapon  systems  in  order  to  apply  the  maximum  potential  of  the  combat 
power  available  at  a  given  space  in  time.  Data  from  the  operational  environment  is  considered 
along  with  information  about  friendly  and  adversarial  forces  to  provide  a  body  of  knowledge 
from  which  operational  commands  can  base  their  decisions. 

To  achieve  information  superiority,  the  U.S.  military  operates  in  accordance  with  the 
principles  of  Network  Centric  Warfare  (NCW).  NCW  emphasizes  decentralized  battlespace 


16  Mark  Jansen,  Hugo  Trepant,  Abdulkader  Lamaa,  and  Andrew  Suddards,  Achieving  information  superiority:  Five 
imperatives  for  military  transformation,  (Booz  and  Company  report,  2014). 

17  Col  William  Hartig,  Thinking  about  Complexity,  (Newport,  RI:  USNWC,  2012).  See  also  David  S.  Alberts  and 
Richard  E.  Hayes,  Power  to  the  Edge:  Command,  Control,  in  the  Information  Age,  (Washington,  D.C.:  DoD 
Command  and  Control  Research  Program,  2005). 

18  Marine  Corps  Doctrinal  Publication  (MCDP)  1-3,  Tactics,  (Department  of  the  Navy,  Washington,  D.C.,  1997). 
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information  collection,  analysis,  and  dissemination  to  and  from  the  forward  edges  of  combat. 19 
Network  Centric  Warfare  focuses  on  sharing  situational  awareness  amongst  military  forces  in  a 
collaborative  manner.  This  collaboration  is  intended  to  synchronize  distributed  efforts  such  that 
dispersed  combat  power  can  be  focused  at  the  decisive  point  in  a  conflict.  The  principle 
challenge  of  maintaining  accurate  situational  awareness  (SA)  in  a  common  operating  picture  has 
been  largely  delivered  in  modern  battle  management  systems.  The  problem  is  that  much  of  this 
battlespace  SA  remains  locked  within  computers  and  tethered  within  tactical  operating  centers. 
The  next  step  in  enhancing  the  SA  provided  within  NCW  will  come  in  the  migration  of  this 
information  to  smartphones  and  wearable  technologies. 


19  David  S.  Alberts,  John  J  Garstka,  and  Frederick  P  Stein,  Network  Centric  Warfare:  Developing  and  Leveraging 
Information  Superiority  ('Washington,  D.C.:  DoD  C4ISR  Cooperative  Research  Program,  1999). 
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Augmented  Reality  Technology  Review  -  Smartphones,  HUDs,  and  HMDs 


The  following  sub-sections  will  provide  descriptions  of  various  forms  of  AR 
technologies.  Though  the  true  beginning  of  AR  began  with  the  development  of  heads-up- 
displays  (HUD)  the  recent  explosion  in  commercial  AR  applications  has  stemmed  from  their  use 
in  smartphones  and  mobile  devices.  As  such,  this  section  will  begin  by  describing  this  trend  and 
explaining  what  it  portends  for  future  AR  system  development. 

Smartphones 

The  first  commercially  viable  AR  devices  were  smartphones.  In  2007,  roughly  20%  of 
global  customers  had  a  smartphone.  Today  in  2015,  over  80%  of  first  world  nations  do  with 
other  countries  close  behind.”  Smartphones  allow  users  to  unplug  from  their  offices  and  homes 
and  effortlessly  begin  to  merge  digital  content  with  real-life  experiences.  A  smartphone  provides 
its  users  a  variety  of  capabilities.  The  most  obvious  use  for  smartphones  concerns  the  ubiquitous 
access  to  communication,  whether  it  is  voice,  video,  or  chat.  However,  it  is  the  access  to  the 
limitless  catalogue  of  digitized  knowledge  and  information  processing  that  has  had  such  a 
transformative  impact  and  kick-started  the  development  of  AR  applications.  Smartphones,  like 
computers  before  them,  offer  an  externalized  means  of  cognition.  However,  unlike  computers, 
smartphones  enable  their  users  to  interact  with  this  content  everywhere  they  have  access  to  a 
wireless  signal.  Augmented  reality  applications  leverage  this  access  to  content  and  external 
processing  to  deliver  digital  data  meant  to  enhance  one’s  understanding  of,  and  interaction  with 
the  real-world. 


20  Gartner  historical,  Jefferies  and  Company,  Inc.  “Exhibit  1:  Smartphone  Penetration  Estimates,” 
(http://zeendo.com/info/smartphone-users-bv-countrv/,  accessed  October  11,  2015). 
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Smartphones  have  transformed  how  people  interact  with  their  environments  in  a  number 
of  ways.  First,  they  have  made  it  easier  to  navigate  to  desired  locations.  Smartphones  have  GPS 
sensors  that  track  their  location  and  navigation  applications  that  help  users  plan  routes  based  on 
this  information.  Today  smartphone  users  always  know  how  to  get  to  their  desired  location. 

Next,  smartphones  allow  users  to  quickly  query  online  content  to  answer  questions  and  fill  in 
gaps  in  their  understanding.  Whether  it’s  a  date  in  history  or  the  name  of  a  forgotten  actor, 
smartphone  users  can  quickly  find  answers  through  an  online  search,  thereby  expanding  their 
understanding.  In  addition,  smartphones  provide  access  to  decision-making  aids.  For  example, 
when  determining  whether  a  product  sold  in  a  store  is  a  fair  price,  many  shoppers  will  frequently 
use  their  smartphones  to  compare  prices.  Furthermore,  many  consumers  use  their  smartphones  to 
research  reviews  and  ratings  for  products  and  services.  Many  shoppers  will  quickly  look  up  a 
product’s  rating  by  other  consumers  before  they  decide  to  make  a  purchase.  They  will  also 
examine  ratings  about  the  quality  of  experiences  -  such  as  dining  at  a  restaurant,  the  quality  of 
entertainment,  or  the  condition  of  a  hotel  and  its  services.  By  providing  navigational  tools, 
enriched  understanding,  and  expert  advice,  today’s  smartphones  have  already  begun  to  augment 
their  users’  sense  of  reality. 

Smartphones  also  placed  a  multitude  of  digital  sensors  in  the  hands,  and  on  the  bodies,  of 
their  users,  thereby  augmented  their  sensory-perceptual  awareness.  Through  contextually  aware 
applications,  the  data  from  these  sensors  provides  environmental  feedback  to  their  users.  This 
sensory-perceptual  expansion  has  generated  novel  means  to  enhance  the  understanding  of  users’ 
environments  and  themselves.  For  example,  biofeedback  applications  can  sense  a  user’s 
temperature,  and  heartrate,  and  motion-state  in  order  to  estimate  changes  in  activity  levels  and 
the  corresponding  caloric  use.  These  sensors  are  also  being  employed  in  the  workplace. 
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Engineers  are  increasingly  turning  to  their  smartphones  to  test  products  and  equipment  through 

applications  that  use  the  sensors  native  to  these  devices  to  create  tachometers,  hygrometers, 

2 1 

seismographs,  frequency  analyzers  and  other  instruments. 

The  sensor  data  from  individual  smartphones  is  also  being  combined  with  the  data  from 
other  mobile  devices  and  external  data  sources  to  provide  a  larger-scale  of  shared  awareness 
known  as  crowd-sensing.”  For  example,  the  relative  speed  of  a  collection  of  smartphones 
traveling  in  cars  on  the  same  stretch  of  road  can  be  collated  to  measure  traffic  patterns.  Likewise, 
companies  like  Open  Signal,  entice  users  to  participate  in  their  crowd-sensing  enterprise  by 
allowing  them  to  use  their  application  to  find  where  the  best  free-Wi-Fi  signals  reside  while  their 
devices  begin  to  collect  and  share  Wi-Fi  sensor  data  from  their  device  back  to  Open  Signal’s  user 
base.  Crowd-sensing  and  crowd-sourcing  smartphone  applications  have  also  been  developed 
for  other  spatio-temporal  analytical  forecasts.  Weather  forecast  applications  can  combine  sensor 
data  from  dispersed  smartphones  to  provide  more  accurate  measurements  of  changing 
meteorological  conditions.  Similarly,  government  health  services  can  monitor  the  mobility 
behavior  pattern  of  individuals  traveling  two  and  from  clinics  and  hospitals  to  predict  potential 
disease  outbreaks  or  other  emergencies.  The  measured  convergence  of  smartphones  can 
indicate  flocking  or  queuing. 

When  combined  with  details  about  smartphone  users’  internet  search  history, 
communication  patterns,  or  social  network  connections,  this  data  can  become  invaluable  in 


21  Kathryn  Alexander,  “Instruments:  Sensors  and  Apps  are  Turning  Smartphones  into  Engineering  Measurement 
Tools,”  (Mechanical  Engineering,  September  1,  2015). 

22  Moo-Ryong  Ra,  Bin  Liu,  Tom  La  Porta,  Ramesh  Govindan,  “Medusa:  A  Programming  Framework  for  Crowd- 
Sensing  Applications,”  Mobisys’  12,  Low  Wood  Bay,  Lake  District,  UK,  June  25-29,  2012) 

2’  Paul  Sawers,  “How  OpenSignal  uses  Millions  of  Smartphones  to  Create  a  Giant  Social  Library,”  (VentureBeat, 
January  30,  2015,  http://venturebeat.com/2015/01/30/how-opensignal-uses-millions-of-smartphones-to-create-a- 
giant-social-laboratory/,  accessed  August  12,  2015). 

24  Paul  Lukowicz,  Alex  Pentland,  and  Alois  Ferscha,  “From  Context  Awareness  to  Socially  Aware  Computing,” 
(Pervasive  Computing,  IEEE  CS,  Jan-Mar,  2012),  p.  37. 
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understanding  and  predicting  behavior.  Corporations  and  governments  desire  to  have  this 
information  -  one  for  profit  the  other  for  security.  This  raises  a  number  of  civil  liberties 
concerns.  Despite  various  security  and  privacy  regulations  in  place  to  limit  exactly  what  an 
organization  can  obtain  from  users’  smartphone  sensor-data  -  many  users  unwittingly  allow  for 
this  information  to  be  exchanged  when  they  enable  location-services  to  operate.  Time  will  tell 
whether  most  are  willing  to  forgo  these  concerns  for  the  benefits  they  receive  by  participating  in 
crowd-sourcing  applications.  It  appears  that  the  trend  is  moving  away  from  individual  privacy 
toward  shared-awareness. 

In  addition  to  the  AR  innovations  of  improved  sensory-perceptual  experiences  of  the  self 
and  collective,  the  next  advancements  in  enhancing  real-world  experiences  comes  in  the  form  of 
automated  personal  assistants.  These  personal  assistants  are  now  available  in  all  major 
smartphone  operations  systems.  Programs  like  Apple’s  Siri,  Google’s  Now,  and  Microsoft’s 
Cortina  respond  to  user’s  directions  and  questions,  much  like  one  would  expect  a  human 
secretary  or  personal  assistant  would.  These  semi-artificially  intelligent  applications  are 
accelerating  the  human-AR  symbiotic  relationship.  No  longer  do  people  have  to  depend  only  on 
their  experience  and  understanding  of  how  to  interact  with  this  brave  new  digital  world  -  they 
can  ask  a  digital  companion  for  help.  These  assistants  are  beginning  to  learn  their  users’ 
behaviors  and  predict  what  they  will  need  and  when  they  need  it.26  They  are  also  beginning  to 
direct  the  behavior  of  their  users  rather  than  waiting  to  be  directed.  Through  assigned  reminders, 
context-awareness,  and  object  recognition,  automated  assistants  can  alert  their  users.  These 
prompts  may  help  the  forgetful  or  distracted,  but  they  also  can  become  an  auto-pilot  feature  for 

25  Rahel  Gebreyes,  “Research  Shows  Location  Tracking  Apps  Like  Grinder  Could  Threaten  Your  Privacy,”  (Huff 
Post,  January  28,  2015,  http://www.huffingtonpost.com/2015/01/28/location-tracking-grindr- 

privacy  n  6564864.html,  accessed  October  24,  2015). 

26  Alex  Fitzpatrick,  “Why  We  Finally  Have  the  First  True  ‘Smartphones’,”  (Time.com,  June  12,  2015). 
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personal  every-day  routines.  Smartphone  users  access  to  these  types  of  virtual  personal  assistants 
is  clearly  augmented  their  real-life  experiences  and  behaviors. 

Access  to  these  AR  capabilities,  along  with  the  myriad  of  other  benefits  of  smartphones 
has  made  them  an  indispensable  part  of  everyday  life  for  many.  In  fact,  some  would  rather  leave 
home  today  without  their  wallets,  then  without  their  smartphones.  The  proliferation  of 
smartphones,  with  their  various  AR  applications,  has  demonstrated  the  potential  of  AR  systems 
to  enhance  experiences  and  improve  users’  interactions  with  their  environments.  The  user 
experience  of  AR  applications  with  smartphones,  however,  leaves  much  to  be  desired.  As  a 
result,  AR  devices  are  being  designed  to  provide  uninterrupted  AR  experience  for  users.  Unlike 
smartphones  that  move  from  pockets  to  hands  to  eyes  -  they  are  worn  as  glasses,  visors,  or 
contact  lenses.  As  such,  they  provide  continuous  access  to  AR  content.  With  their  arrival,  AR 
systems  will  fundamentally  change  how  users  interact  with  each  other  and  their  surroundings  - 
just  as  smartphones  have  done  over  the  past  decade. 

Heads-Up-Displays  (HUD) 

Heads-Up-Displays  (HUD)  were  the  first  AR  systems  developed  to  present  digital  data 
within  a  user’s  field-of-view  (FOV).“  Field-of-view  AR  devices,  like  a  HUD,  provide  a 
mediated  form  of  reality  by  overlaying  computer  generated  information  on  users’  visual 
perceptual  experience.  Augmented  reality  systems  achieve  this  by  combining  the  virtual  with  the 
real.  It  registers  and  processes  sensor  data  from  a  user’s  environment  so  it  can  produce  and 

27  Meena  H.  Duerson,  “We’re  Addicted  to  Our  Phones:  84%  Worldwide  Say  They  Couldn’t  Go  a  Single  Day 
Without  Their  Mobile  Device  in  Their  Hand,  ”  (August  16.  Accessed  May  15,  2015. 

http://www.nvdailvnews.com/life-stvle/addicted-phones-84-worldwide-couldn-single-day-mobile-device-hand- 

article- 1.1 137811,  2012). 

28  Mark  A.  Livingston,  Lawrence  J.  Rosenblum,  Dennis  G.  Brown,  Gregory  S.  Schmidt,  Simon  J.  Julier,  Yohan 
Baillot,  J.  Edward  Swan  II,  Zhuming  Ai,  and  Paul  Maassel,  “Military  Applications  of  Augmented  Reality,”  (Naval 
Research  Laboratory,  Washington,  D.C.,  2011). 
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anchor  computer  generated  information  relevant  for  a  given  setting.  It  merges  digital  content 
with  users’  perceptual  experience  by  rendering  2D  and  3D  graphics  over  their  visual  FOV  as 
they  move  through  and  interact  with  their  surroundings.  This  experience  differs  from  Virtual 
Reality  (VR)  which  generates  fully  immersive  virtual  environments  in  which  users  interact  with 
virtual  content.  Augmented  reality  systems,  on  the  other  hand,  are  designed  for  use  in  the  real- 
world.  The  earliest  forms  of  FOV  AR  systems  were  developed  for  fighter  pilots  in  the  form  of 
Heads-Up-Displays  (HUD).29 

The  military  has  continued  to  invest  in  the  development  of  HUDs  because  it  was  found 
that  pilots  who  used  HUDs  operated  their  aircraft  more  ably  than  when  they  relied  on 

on 

conventional  flight  instruments  alone.  A  HUD  in  an  aircraft  employs  a  semi-transparent 
combiner  located  between  the  pilot’s  eyes  and  the  aircraft  windshield.  AR  information  is 
projected  onto  the  combiner  enabling  the  pilot  to  view  both  the  digital  content  and  the  real-world 
scene.  The  first  commercially  available  HUDs  were  designed  to  help  civilian  pilots  during 

o  1 

landings.  Today’s  aircraft  HUDs,  however,  include  information  about  airspeed,  altitude, 

IT 

elevation,  flight  path,  navigation,  as  well  as  automatic  flight  control  system  information. 

Aircraft  are  not  the  only  vehicles  to  employ  AR  HUD  capabilities. 

Augmented  reality  HUDs  have  also  been  developed  for  watercraft  and  the  maritime 
environment.  The  use  of  these  systems  provided  increased  situational  awareness  and  improved 

O'? 

precision  piloting/  The  Office  of  Naval  Research  supported  the  development  of  the  Augmented 
Reality  for  the  Common  Operational  Picture  (ARVCOP)  system.  This  AR  system  tracks  the 


Gregory  Kipper  and  Joseph  Rampolla,  Augmented  Reality:  An  Emerging  Technologies  Guide  to  AR.  (Waltham, 
MA:  Elsevier,  2013). 

30  Robert  B.  Wood  &  Peter  J.  Howells,  “Head-Up  Displays,”  (The  Avionics  Handbook,  CRS  Press,  LLC,  2001). 

31  Ibid. 

32  Ibid. 

33  Thomas  Zysk,  Jeffory  Luce,  and  James  Cunningham,  “Augmented  Reality  for  Precision  Navigation:  Enhancing 
Performance  in  High-Stress  Operations,”  (GPSWorld.com,  June  1,  2012, 

http://gpsworld.com/defensenavigationaugmented-reality-precision-navigation-13032/),  pp.  47-51. 
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precise  location  of  a  vessel  and  then  projects  geographically  registered  navigation  overlays  for 
pilots  to  see  as  they  navigate  a  pre-planned  route.  During  evaluations,  the  use  of  this  system 
helped  reduce  errors  from  translating  2D  map  data  to  the  pilot’s  3D  perspective.34  When 
maritime  pilots  used  the  ARVCOP  to  perform  precision  piloting  tasks  they  navigated  more 
precise  routes  along  planned  routes.  Operators  using  ARVCOP  also  reported  they  were  more 
confident  in  maneuvering  in  unfamiliar  waters,  at  higher  rates  of  speed,  as  well  as  at  night  or  in 
inclement  weather.  These  benefits  have  also  found  their  way  from  the  water  to  the  road. 

Automotive  manufacturers  such  as  BMW,  Mercedes-Benz,  Mitsubishi,  and  SAAB  have 
recently  made  investments  in  developing  HUD  like  windshields.  These  innovations  aim  to  keep 
driver’s  attention  focused  on  the  road  ahead  by  displaying  traditionally  dash-mounted 
information  in  their  FOV.36  This  information  includes  navigation  route  guidance,  vehicle-to- 
vehicle  safety  alerts  (i.e.  warnings  about  proximity  to  other  vehicles),  as  well  as  signals  for  lane 
departures  and  traffic  alerts.  It  may  soon  include  AR  aids  for  driving  during  zero- visibility 
conditions  (i.e.  when  fog,  rain,  or  snow  occludes  vital  visual  navigational  information).  An 
onboard  computer  will  use  the  car’s  laser  and  radar  sensors,  and  cross  reference  available  3D 

•50 

maps  of  an  area,  to  project  AR  driving  cues  on  the  car’s  windshield.  This  AR  information 
provides  indispensable  information  to  drivers  without  requiring  them  to  look  away  from  the 
environment. 


34  Ibid,  p.  49. 

35  Ibid,  p.  48. 

36  Nick  Jaynes,  “Forget  head-up  displays.  Mitsubishi's  concept  has  an  augmented  reality  windshield,”  (Mashable, 
October  9,  2015,  http://mashable.eom/2015/10/09/mitsubishi-ex-concept/#CA  HwUPo5uqu,  accessed  October  11, 
2015.) 

37  Ibid. 

3S  Alex  Oagana,  “Three  Future  Safety  Technologies  That  Could  Change  Driving  as  we  Know  it,”  (Autoevolution, 
October  7,  2015,  http://autoevolution.com/news/three-future-safetv-technologies-that-could-change-driving-as-we- 
know-it-100779.html,  accessed  October  11,  2015). 
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The  military  has  also  researched  AR  HUD  capabilities  for  ground  vehicles.  Like  aircraft 
and  commercial  vehicles,  military  vehicle  crews  can  also  benefit  from  the  navigational  aids  and 
enhanced  situational  awareness  provided  by  AR  systems  integrated  with  a  vehicle’s  sensors. 
However,  increased  calls  for  the  use  of  AR  systems  has  resulted  from  the  increasing  dangers  to 
exposed  vehicle  crew  members  and  the  growth  of  remotely  controlled  weapon  systems.  One 
vehicle  AR  HUD  system  available  for  military  use  was  developed  by  SAAB.  This  AR  system 
employs  components  that  include  a  Local  Situational  Awareness  System  (LSAS),  Driver  Vision 
Enhancement  System  (DVES),  and  a  Vehicle  Camera  System  (VCS)  to  provide  high-resolution 
images,  and  sensor  data,  with  low  latency  to  drivers  and  their  crew.40  These  vehicular 
components  can  provide  360  degree  of  the  vehicle’s  exterior,  IR  cameras  for  when  climate 
conditions  occlude  optical  cameras,  and  the  connection  and  control  of  weapons  and  their  sights.41 
They  are  also  intended  to  link  to  other  battlefield  management  systems  so  that  commanders  can 
understand  the  location  and  disposition  of  their  forces.  ~  These  systems  are  useful,  but  they  still 
require  operators  to  remain  in  relatively  fixed  positions  so  they  can  view  mounted  HUD  systems. 

Head-Mounted-Displays  (HMD) 

As  the  name  implies,  Head-Mounted-Displays  (HMD)  are  AR  systems  that  a  person 
wears  on  his  or  her  head.  They  include  helmets  with  visors,  various  glasses,  as  well  as  contact 
lenses.  These  display  devices  liberate  users  from  HUDs  by  allowing  them  the  freedom  to  move 
their  heads  without  losing  sight  of  AR  content.  They  also  allow  operators  to  move  from  beyond 
cockpits  out  into  the  real-world.  They  can  do  this  because  they  precisely  track  the  orientation  and 

39  Brian  Kindamo,  “Vehicle  Situational  Awareness,”  (MilTech,  Vol.  1,  2015),  p.  48-49. 

40  Ibid,  p.  49. 

41  Saab,  “Rugged  Vetronics  Systems,”  (Saab  Corporate  Brochure,  http://saab.com/globalassets/publications- 
pdfs/support-and-services/militarv-electronics-solutions/234-0735  rvs  broschure  a4  update  k5  low. pdf,  accessed 

October  11,  2015). 

42  Ibid,  p.  5. 
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location  of  the  user’s  head  and  body  and  synchronize  this  infonnation  with  virtual  information 
rendered  to  a  user’s  visual  FOV. 


Head-Mounted-Displays  can  be  categorized  in  a  number  of  manners  -  but  a  common 
distinction  concerns  the  manner  in  which  information  may  be  visually  presented.43  Three  basic 
categories  of  HMDs  exist.  Monocular  HMDs  display  information  to  a  single  eye.  Biocular 
HMDs  display  information  to  both  eyes,  however,  this  information  comes  from  a  single  sensor 
and  therefore  does  not  necessarily  provide  depth  cues  or  perspective.  Binocular  HMDs,  however, 
present  two  visual  images,  one  for  each  eye  but  from  sensors  that  provide  3D  perspective.44 
Beyond  these  three  categories  the  manner  in  which  information  is  displayed  can  also  differ. 
Traditionally,  HMD  had  AR  information  relayed  optically  and  then  overlaid  on  reflective  and 
refractive  lens  elements.45  Another  feature  of  many  HMDs  is  their  ability  to  track  eye- 
movements  so  as  to  identify  the  object  of  a  user’s  attention  and  to  project  AR  information 
accurately  and  in  the  correction  orientation  with  respect  to  a  user’s  visual  perspective.46  HMDs 
can  render  virtual  information  in  a  number  of  ways.  For  instance,  one  HMD  technology  can  use 
lasers  to  project  an  image  directly  onto  the  retina  of  a  user’s  eyes.47  Another  design,  developed 
by  BAE  Systems  employs  holographic  optics  presented  between  two  optic  plates  to  merge  real- 
world  information  with  virtual  content.  As  digital  technologies  increase  in  capacity,  and 
decrease  in  size,  they  will  continue  to  enable  light-weight,  high-resolution,  high-speed  HMDs. 


43  Michael  M.  Bayer,  Clarence  E.  Rash,  James  H.  Brindle,  “Introduction  to  Helmet -Mounted  Displays,”  (in.  Helmet- 
Mounted  Displays:  Sensation,  Perception  and  Cognition  Issues,  Ed.  Clarence  E.  Rash,  Michael  B.  Russon,  Tomasz 
R.  Letwoski,  Elmar  T.  Schmeisser,  U.S.  Army  Aeromedical  Research  Laboratory,  Ft.  Rucker,  AL,  2009),  pp.  47- 
107. 

44  Ibid,  p.  49. 

45  Ibid,  p.  50. 

46  Jannick  Rolland  &  Hong  Hua,  “Head-Mounted  Display  Systems,”  (Encyclopedia  of  Optical  Engineering,  Taylor 
&  Francis,  2005). 

47  Michael  M.  Bayer,  et  al.,  “Introduction  to  Helmet-Mounted  Displays,”,  p.  52. 

48  Ibid,  p.  53. 
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Original  HMDs  were  developed  for  military  aircraft  pilots,  due  in  large  part  to  the  same 
set  of  motivations  that  led  to  the  development  of  cockpit  HUDs.  The  military  aviation 
community  began  to  research  and  develop  HMD  systems  in  the  1970s  for  both  rotary  and  fixed- 
wing  aircraft.  The  wide  field-of-regard  provided  by  HMDs  allows  pilots  and  crews  to  observe  far 
more  of  their  external  surroundings  while  still  receiving  system  information  through  the  HMD. 
Beyond  this  fear,  and  the  navigation  capabilities  typical  of  earlier  HUDs,  most  modem  aviation 
HMDs  also  included  weapon  sighting  information.  Access  to  this  information  produced  a 
significant  reduction  in  the  time  required  to  employ  weapons  against  a  target.49  Furthermore, 
with  the  addition  of  advanced  sensors  in  these  aircraft,  such  as  infrared  and  thermal  optics,  pilots 
using  HMDs  are  able  to  see  AR  data  merged  with  these  enhancing  images.  All  of  this  additional 
information  increases  the  ability  of  pilots  and  their  crews  to  remain  oriented  to  their 
surroundings,  and  employing  their  weapon  systems,  while  continuing  to  pilot  their  aircraft.  This 
allows  pilots  and  crews  to  maintain  SA  and  allows  them  to  more  rapidly  orient  toward,  and 
remain  fixated  on  adversaries. 

The  Generation  3  Helmet-Mounted  Display  System  (HMDS)  for  the  F-35  Joint  Strike 
Fighter  (JSF)  represents  that  latest  assembly  of  AR  HMD  technologies.  The  JSF  helmet  was 
designed  to  produce  a  symbiotic  type  relationship  between  the  pilot  and  the  aircraft  itself.50  One 
of  the  unique  AR  capabilities  of  the  JSF  helmet  is  that  it  allows  pilots  to  see  through  the  aircraft. 
When  they  look  down,  they  see  the  world  below.  When  they  look  back,  they  see  the  horizon 
behind  them.  Six  cameras  are  embedded  in  the  skin  of  the  F-35  capturing  and  relaying  their 
imagery  to  the  pilot’s  helmet.  The  visor  in  the  helmet  acts  like  an  HMD  that  presents  traditional 


49  Ibid,  p.  56. 

50  Christian  Davenport,  “Meet  the  Most  Fascinating  Part  of  the  F-35:  The  $400,000  Helmet,”  (The  Washington  Post, 
April  1,  2015,  https://www.washingtonpost.com/news/checkpoint/wp/2015/04/01/meet-the-most-fascinating-part-of- 
the-f-35-the -400000-helmet/,  accessed  October  11,  2015. 
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navigational  information,  along  with  other  battle  management  system  data.  F-35  pilots  can  track 
the  location  of  friendly  and  adversarial  forces  in  their  helmet’s  display.  Despite  its  costs,  and 
delays  in  production  due  to  technical  challenges,  this  new  helmet  stands  as  the  most  advanced 
AR  HMD  ever  produced. 

Modern  HMDs  have  also  been  developed  for  military  land  vehicles.  As  previously  noted, 
maintaining  situational  awareness  from  within  an  armored  vehicle  or  tank  can  be  a  challenge. 

In  a  desire  to  allow  the  operators  and  crews  of  military  land  vehicles  greater  situational 
awareness  and  freedom  of  movement  within  their  vehicles  BAE  has  developed  an  HMD  AR 
system  for  armored  vehicles  and  tanks.51  BAE  Systems’  BattleView  360  system  builds  upon  the 
efforts  it  made  in  developing  a  backup  F-35  HMDS  when  it  appeared  that  the  primary  contractor, 
Rockwell  Collins,  may  not  be  able  to  deliver  a  viable  device.  “  BAE  Systems’  BattleView  360 
provides  a  common  operating  picture  by  presenting  and  tracking  the  positions  of  surrounding 
features  of  interest  in  2D  and  3D  modalities.  This  system  inputs  data  from  various  optical  and 
infrared  sensors  on  the  vehicle  and  merges  this  information  with  digital  map  data.  Operators  then 
use  HMD  interfaces  and  touchscreen  units  within  vehicles  to  visualize  this  information.54  The 
employment  of  AR  HMD  systems  within  military  vehicles  has  demonstrated  the  exciting 
potential  of  these  technologies  to  increase  the  SA  of  their  crews  and  the  overall  effectiveness  of 
these  combat  systems. 


51  Anthony  Wood,  “BAE  Systems’  BattleView  360  is  Made  to  Raise  Situational  Awareness  in  Tank  Crews,” 
(Gizmag,  September  22,  2015,  http://www.gizmag.com/bae-systems-battleview-360/39513/.  accessed  October  11, 
2015). 

52  Lara  Seligman,  “Rockwell  Delivers  First  Gen  3  Helmet  for  F-35,”  (Defense  News,  August  12,  2015, 
http://www.defensenews.com/storv/defense/air-space/strike/2015/Q8/12/rockwell-delivers-first-gen-3-helmet-f- 

35/31538101/.  accessed  October  1 1,  2015). 

53  Ola  Thoren,  “BattleView  360  ‘See-through’  Armored  Vehicle  System  Makes  DSE1  Debut,”  (BAE  Systems  Public 
Affairs,  September  18,  2015, 

http://www.baesystems.com/cs/Satellite?c=BAENews  Release  C&childpagename=US%2FBAELayout&cid=1434 

566341731&pagename=US Wrapper,  accessed  October  11,  2015). 

34  Anthony  Wood,  “BAE  Systems’  BattleView  360  is  Made  to  Raise  Situational  Awareness  in  Tank  Crews,” 
(GizMag,  September  22,  2015,  http://www.gizmag.com/bae-systems-battleview-360/39513/,  accessed  October  11, 
2015). 
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The  success  of  HMDs  used  within  modem  military  vehicles  has  increased  calls  for 
dismounted  AR  HMD  systems.  In  2009,  DARPA  began  a  project  called  ULTRA- VIS  (Urban 
Leader  Tactical  Response,  Awareness  &  Visualization),  in  collaboration  with  Lockheed  Martin 
and  Microvision  to  develop  an  HMD  for  ground  forces.55  This  effort  sought  to  provide  AR 
overlays  within  a  tactical  optic  dismounted  personnel  could  wear  on  their  helmets.  The  AR 
software  behind  ULTRA-VIS,  ARA’s  ARC  4,  provides  semi-transparent  graphical  overlays 
depicting  blue  forces,  hazards,  imagery,  navigation  routes,  and  alerts.56  It  accepts  inputs  from  a 
sensor  module  of  cameras,  as  well  as  GPS  data,  head-tracking  data,  and  then  fuses  this 
information  to  render  digital  content  in  concert  with  a  user’s  location  and  their  FOV 
perspective.57  The  biggest  challenge  the  ARC  4  software  had  to  overcome  was  tracking  a  user’s 
precise  location  as  well  as  their  eye  movements  as  they  moved.  This  data  enables  the  ARC  4 
software  to  render  digital  content  over  a  users’  real-world  view  and  maintain  its  location  and 

CO 

correction  orientation.  In  2014,  DARPA  ended  its  ULTRA- VIS  program,  however  ARA  has 
continued  to  develop  its  ARC  4  software,  and  others  have  continued  to  develop  AR  HMDs. 

BAE  System’s  Q-Warrior  display,  which  had  also  been  developed  as  part  of  the  ULTRA- 
VIS  program,  has  shown  the  most  promise  for  military  dismounts.  The  Q-Warrior  display,  is  a 
light-weight,  helmet-mounted,  monocular  AR  optical  device.59  It  attaches  to  a  helmet’s  FAST 


55  Alexis  C.  Madrigal,  “How  DARPA’s  Augmented  Reality  Software  Works,”  (The  Atlantic,  May  27,  2014, 
http://www.theatlantic.com/technology/archive/2014/05/hows-darpas-augmented-realitv-software-works/371652/, 

accessed  October  17,  2015). 

56  Joey  Cheng,  “DARPA-funded  Augmented  Reality  Technology  Moves  to  Commercial  Sector,”  (Defense  Systems, 
May  29,  2014,  https://defensesvstems.com/articles/2014/05/29/ara-augmented-reality.aspx,  accessed  October  16, 
2015). 

57  Rose  Evelth,  “Augmented  Reality  System  Goes  from  Military  to  Market,”  (The  Daily  Beast,  May  27,  2014, 
http://www.thedailvbeast.com/articles/2014/05/27/augmented-realitv-svstem-goes-from-militarv-to-market.html, 

accessed  October  16,  2015). 

5S  Alexis  C.  Madrigal,  “How  DARPA’s  Augmented  Reality  Software  Works,”  2014. 

59  Allen  McDufee,  “At  Last,  A  Google  Glass  for  the  Battlefield,”  (Wired,  February  24,  2014, 
http://www.wired.com/2014/02/battlefield-glass/,  accessed  October  17,  2015). 
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rail  system  and  weighs  about  400  grams.60  The  device  provides  a  high-resolution  transparent 
display  on  which  full-color,  3D  graphics  are  shown.  Though  ARA’s  ARC  4  AR  software  has 
served  as  the  brains  within  the  Q-Warrior  display,  other  AR  software  could  be  developed  to 
control  Q-Warrior.  Likewise,  ARA’s  ARC  4  AR  software  has  been  integrated  with  different 
wearable  displays  besides  BAE  Systems  Q-Warrior,  for  example:  Lumus,  Vusix,  and  Exelis. 
Currently,  despite  a  number  of  efforts  to  field  an  AR  HMD  system  within  each  branch  of  service, 
no  single  software  -  display  device  solution  exists  outright.  Many  are  in  development,  but  the 
field  of  candidate  AR  HMD  systems  remains  broad. 

Augmented  Reality  Visors,  Glasses,  and  Contact  Lenses 

Augmented  reality  glasses  and  AR  contact  lenses  are  another  form  of  an  AR  HMD  -  they 
just  happen  to  be  less  bulky  and  cumbersome.  These  systems  seek  to  scale-down  the  size  and 
complexity  of  AR  systems  from  that  found  in  AR  HMDs  so  that  users  can  wear  these  devices 
and  interact  with  them  intuitively  during  their  daily  lives,  at  home,  in  the  office,  and  everywhere 
in  between.  The  sea-change  in  AR,  and  the  reason  why  so  many  potential  AR  users,  and  analysts, 
are  excited  about  the  future  of  AR  technologies,  is  because  of  the  commercially  viable  AR  smart- 
glasses  currently  coming  to  market.  The  following  section  will  briefly  outline  the  currently 
available  AR  smart-glasses  and  AR  contact  lenses. 

Microsoft  -  HoloLens.  Microsoft  has  dedicated  over  a  thousand  employees  and  a  billion  dollars 
toward  the  development  of  its  AR  product  -  HoloLens.61  HoloLens  is  a  lightweight  head- 


60  BAE  Systems  Q-Warrior  Data  Sheet, 

http://www.baesvstems.com/cs/Satellite?blobcol=urldata&blobkev=id&blobtable=MungoBlobs&blobwhere=  14346 

45945575&ssbinary=true,  accessed  October  17,  2015. 

61  Bruce  Harris,  (Microsoft  HoloLens  technology  evangelist,  interview,  September  18,  2015). 
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mounted  AR  visor  that  displays  HD  holographic  images  binocularly.62  It  uses  both  optical 
cameras  and  infrared  sensors,  originally  developed  for  the  Microsoft  Kinect,  to  register  a  user’s 
surroundings.  It  has  an  internal  holographic  processing  unit  that  combines  this  sensor  data,  with 
AR  application  data,  to  geo-register  and  display  holographic  digital  content.63  It  has  no  dangling 
cords  or  wires.  It  is  currently  meant  for  indoor  use.  It  will  run  Microsoft’s  native  operating 
system  Windows  10.  For  this  reason,  any  Windows  10  compatible  software  should  be  able  to 
work  within  HoloLens.  Content  from  these  applications  can  be  geo-registered  and  anchored  to 
real-world  locations  in  2D  and  3D.  It  also  has  a  large  number  of  AR  applications  specifically 
developed  by  Microsoft  for  use  in  HoloLens.64  Microsoft  has  said  that  one  of  its  primary  focuses 
for  HoloLens  use  would  be  for  enterprise  users.65  In  particular,  organizations  that  design  and 
produce  real  life  objects  -  i.e.  graphic  designers,  engineers,  and  architects  -  could  find 
HoloLens’s  ability  to  help  them  visualize  their  creations  in  3D,  to  scale,  real-world  virtual 
representations  incredibly  useful. 

Sony  -  SmartEyeglass.  Sony  has  developed  a  set  of  AR  glasses  in  SmartEyeglass  that 
provide  digital  content  but  do  not  deliver  an  AR  experience.  This  system  has  an  optical  camera, 
microphone,  noise  suppression  sub-microphone,  Bluetooth  capability,  Wi-Fi  capability, 
compass,  gyroscope,  accelerometer,  and  brightness  sensor.66  It  displays  digital  information  from 
a  tethered  Bluetooth  or  Wi-Fi  Android  phone,  binocularly  in  monochrome  -  green.67 

62  Microsoft,  “HoloLens  Hardware,”  (http://www.microsoft.com/microsoft-hololens/en-us/hardware,  accessed 
October  17,  2015). 

6’  David  Murphy,  “Augmented  Reality  Glasses:  What  You  Can  Buy  Now  (or  Soon),”  (Tom’s  Guide,  October  6, 
2015,  http://www.toms guide. com/us/best-ar-glasses,review-2804.html,  accessed  October  17,  2015). 

64  Ibid. 

65  Max  Slater-Robins,  “Here’s  Everything  We  Know  About  Microsoft’s  Augmented  Reality  Headset  -  HoloLens,” 
(Business  Insider,  October  17,  2015,  http://www.businessinsider.com/microsoft-hololens-augmented-reality- 
headset-roundup-20 1 5- 1 0?r=UK&IR=T,  accessed  October  18,  2015). 

66  Sony,  “SmartEyeglass,”  (https://developer.sonv.com/devices/mobile-accessories/smarteyeglass/,  accessed  October 
18,2015). 

67  T u;a 
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SmartEyeglass  provides  information  such  as  directions,  Tweets,  and  other  information  related  to 
a  user’s  location.  They  even  include  real-time  voice  translations  from  input  audio  that  then 
appears  as  displayed  text.  The  smart-glasses  themselves  are  meant  to  be  less  conspicuous  as  an 
AR  HMD  system,  yet  they  still  have  a  wire  that  dangles  from  them  to  a  Sony  Eyeglass’  control 
pack,  which  seems  obtrusive  to  some  who  have  reviewed  these  devices.68  This  control  pack 
contains  the  touch  controls,  battery,  microphone,  and  speaker  for  the  smart-glasses.69 

Google  -  Glass.  Google  Glass  was  recently  discontinued,  however  Google  does  not 

70 

appear  complete  this  this  project  with  Project  Aura  perhaps  serving  as  a  revival  of  this  effort. 
The  original  device  given  to  developers  offered  a  tiny  mounted  display  that  attached  to  a  pair  of 
standard  glasses.  It  came  with  an  optical  camera,  Bluetooth  capability,  Wi-Fi  capability,  and 
12GB  usable  memory.  It  relied  on  a  connection  to  a  user’s  smartphone  for  many  other  sensor 
inputs.  Inside  its  display,  a  number  of  superimposed  information  screens  displayed  information 
such  as  the  weather,  alerts,  missed  calls,  and  turn-based  navigation  based  on  a  user’s 
smartphone’s  GPS.  Glass  could  use  third-party  Android-friendly  applications  as  well.71  This 
information  was  not  geo-registered  to  a  user’s  environment  and  therefore  acted  as  mediated 
information  and  not  true  AR. 

Recon  Instruments  (an  Intel  company)  -  Recon  Jet  Recon  Jet  is  similar  to  Google  Glass, 
only  sportier.  It  has  an  HD  camera,  infrared  sensor,  Bluetooth  capability,  Wi-Fi  capability,  8 
GBs  of  usable  storage,  built-in  GPS,  gyroscope,  accelerometer,  altimeter,  barometer, 
magnetometer,  and  pressure  sensor.  It  uses  Intel’s  dual-core  processor  and  can  pair  with  a 

6S  Rich  Trenholm,  “Sony’s  Prototype  EyeGlass  Smart  Specs  Eye  up  Google  Glass,”  (CNET,  September  5,  2014, 
http://www.cnet.com/products/sonv-smart-eyeglass/,  accessed  October  18,  2015. 

69  Sony,  2015. 

70  Antony  Maina,  “Remember  Google  Glass?  A  Revival  is  Being  Planned,”  (Small  Business  Trends,  September  22, 
2015,  http://smallbiztrends.com/2015/09/google-glass-proiect-aura.html,  accessed  October  17,  2015). 

71  Murphy,  “Augmented  Reality  Glasses:  What  You  Can  Buy  Now  (or  Soon),”  2015. 

72  Recon  Instruments,  “Recon  JET,”  (http://www.reconinstruments.com/products/iet/,  accessed  October  17,  2015). 
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user’s  smartphone  to  get  caller  ID,  SMS  notifications,  and  social  media  information.  "  The  AR 
display  is  mounted  to  a  pair  of  polarized  sports  frames;  however  unlike  Google  Glass,  it  is 
presented  in  the  bottom  of  the  right  lens  versus  the  top.  Recon  Jet  is  designed  for  athletic  use 
such  as  running  or  biking.  It  delivers  contextually  relevant  information,  but  like  Google  Glass,  is 
does  not  geo-register  and  render  digital  content.  It  has  sports-themed  applications  capable  of 
tracking  metrics  such  as  speed,  pace,  distance,  and  heartrate.74  Though  it  has  a  built-in 

7c 

microphone,  it  cannot  yet  be  controlled  through  voice  commands. 

Optivent  -  Ora- 1 .  Optivent’s  Ora-1  is  similar  to  other  smart-glass  form  factors,  such  as 
Google’s  Glass  and  Recon  Instrument’s  Jet.  Like  these  devices,  it  has  a  small  screen  that 
augments  a  user’s  vision.  It  has  an  optical  camera,  1080p  video  recording,  Bluetooth  capability, 
Wi-Fi  capability,  4GB  usable  memory,  ambient  light  sensor,  and  a  photochromic  lenses.  The  key 
difference  between  the  Ora-1  and  similar  devices,  is  that  its  small  display  screen  can  move.  In 
addition,  its  photochromic  lenses  adjust  to  sunlight.  Furthermore,  the  Ora-1  can  serve  as  a  stand¬ 
alone  Android  device  capable  of  running  Android  applications  without  the  need  to  pair  it  with  a 
smartphone.76 

Vuzix  -  M100  Smart  Glasses.  Vuzix  has  long  been  a  leading  developer  of  AR  glasses.  Its 
M100  Smart  Glasses  comes  in  two  primary  categories  -  one  attached  to  a  pair  of  glasses  and 
another  that  are  built  into  a  pair  of  safety  goggles.77  It  comes  with  an  optical  camera,  1080p 
video,  Bluetooth  capability,  Wi-Fi  capability,  4GB  usable  memory,  and  built-in  GPS.  Vuzix  has 
developed  its  own  apphcations  for  the  Ml 00  Smart  Glasses,  but  Andoid  applications  can  also  be 


73  Ibid. 

74  Ibid. 

75  Murphy,  “Augmented  Reality  Glasses:  What  You  Can  Buy  Now  (or  Soon),”  2015. 

76  Murphy,  “Augmented  Reality  Glasses:  What  You  Can  Buy  Now  (or  Soon),”  2015. 

77  Vuzix,  “Background  Information,”  (Vuzix  Product  Website,  2014, 
https://www.vuzix.com/news/background  information/,  accessed  October  17,  2015). 
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used.  It  comes  pre-loaded  with  an  audio-recorder,  camera,  photo  gallery  app,  and  eyesight- 
based  bar  code  scanner.  The  Ml 00  Smart  Glasses  can  be  controlled  through  voice  interactions  or 
by  tapping  various  hardware  buttons  on  the  device,  or  through  an  Android  phone  with  a  virtual 
mouse  and  keyboard.79 

Meta  -  Meta  1 .  Meta  developed  an  AR  visor  similar  in  size  and  shape  to  Microsoft’s 
HoloLens.  Like  HoloLens,  Meta  1  is  meant  for  indoor  use.  It  has  an  optical  camera,  a  3D  depth 
camera,  integrated  3D  Dolby  audio,  360-degree  head-tracking,  9-axis  inertial  measurement  unit, 
compass,  gyroscope,  accelerometer,  and  HDMI  input.80  This  system  is  designed  to  fill  the  FOV 
of  users  -  thereby  engrossing  a  user’s  visual  experience.  Meta  1  can  sense  and  track  a  user’s 
hand  movements,  which  in  turn  can  allow  its  users  to  use  their  hands  to  interact  with  virtual 
content  -  grabbing  and  moving  virtual  objects  and  moving  and  rotating  them  in  space.  Meta 
has  also  recently  announced  the  development  of  Meta  Pro  -  a  sleeker  pair  of  glasses  that  appear 
more  like  Ray-Ban  sunglasses  than  AR  glasses. 

Epson  -  Moverio  BT-200  and  BT-2000.  Epson  developed  its  Mo verio  BT-200  and  BT- 
2000  for  true  AR  experiences.  Both  devices  have  a  depth-sensing  optical  camera,  Bluetooth 
capability,  Wi-Fi  capability,  8  GB  internal  memory,  removable  memory  microSDHC  card,  3D 
Dolby  audio,  compass,  gyroscope,  and  accelerometer.  '  Both  systems  also  rely  on  a  control  unit 
tethered  to  the  display  visor.  They  are  binocular  with  two  screens  display  a  960  x  540-pixel 
image  over  each  eye  of  a  user.83  The  BT-200  supports  gesture  recognition  and  3D  mapping  of  a 
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user’s  surroundings.  These  devices  are  Android  compliant.  The  BT-2000  differs  from  the  BT- 
200  in  that  it  also  has  industrial  grade  IMU  sensor  and  a  more  ruggedized  design.  Epson 

Of 

designed  the  BT-200  for  industrial  workplaces. 

GlassUp.  Unlike  all  of  the  previous  smart-glasses  described,  GlassUp  does  not  have  an 
optical  camera,  nor  does  it  have  multicolor  display  capability.  Instead,  users  get  digital  content 
displayed  only  to  the  center  of  these  glasses.86  Emails,  text  messages,  social  media  updates  all 
pop  up  in  the  center  of  a  user’s  view,  remain  for  a  few  seconds,  and  then  disappear.  It  has 
Bluetooth  capability,  an  accelerometer,  compass,  ambient  light  sensors,  precision  altimeter  and  it 

oo 

is  controlled  via  a  touch-pad.  GlassUp ’s  developers  designed  their  system  to  provide  simple 

on 

aids,  and  not  for  entertainment,  nor  semi-immersive  virtual  experiences. 

Google  Contact  Lenses.  Recently,  Google  filed  a  patent  for  a  wearable  communication 
device  saturated  with  sensors,  memory,  a  microprocessor,  solar-power  cells,  and  shaped  like  a 
contact  lens.90  The  lens  would  be  powered  by  the  solar-power  cells  which  would  harvest  energy 
from  the  sun  and  other  light  signals.  The  contact  lens  could  relay  information  it  can  chemically 
detect,  such  as  a  user’s  body  temperature,  blood- alcohol  level,  exposure  to  allergens  or  other 
airborne  materials  back  to  an  external  device.91  Google’s  patent  also  adds  that  the  lens  would  be 
capable  of  communicating  with  computers  or  mobile  devices,  and  could  potentially  read 
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information  from  various  objects  within  a  user’s  FOV.  ~  Should  this  technology  come  to  fruition, 
it  would  create  the  most  symbiotic  relationship  yet  between  AR  digital  information  and  a  user. 

In  summary,  the  range  of  commercially  viable  AR  smart-glasses  has  never  been  larger. 
Their  form  factors  and  use-cases  range  widely  from  inconspicuous,  light-weight  glasses  to  bulky 
head-mounted  visors.  Some  are  meant  for  niche  markets  such  as  elite  sports  performance 
monitoring  and  industrial  maintenance  aids;  others  are  intended  for  broader  audiences.  As  varied 
as  these  products  and  their  uses  are,  they  share  similarities  in  their  fundamental  engineering.  The 
following  section  describes  a  generic  AR  system  architecture  that  many  AR  systems  employ. 
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How  Augmented  Reality  Systems  Work 


Augmented  reality  systems  encode  detailed  information  about  a  user’s  environment  and 
their  actions  there  within.  Data  inputs  from  the  environment  provide  information  to  AR  mapping 
applications  that  encode  details  about  a  user’s  surroundings  and  the  locations  of  objects  within 
space.  This  mapping  data  helps  generate  a  virtual  map  that  the  AR  system  will  later  use  to  anchor 
virtual  content  in  2D  and  3D  perspectives.  In  addition  to  building  a  virtual  map,  these  data  inputs 
can  also  be  used  by  other  AR  applications.  Some  AR  applications  can  perform  object 
recognition.  For  example,  the  retail  sector  has  invested  heavily  in  developing  such  applications 
so  it  can  enhance  the  shopping  experience  of  customers.  Once  an  object  is  recognized  in  retail 
setting,  this  data  can  cue  enhancing  information  such  as  the  name  of  the  object,  its  price,  or 
perhaps  reviews  of  the  object.  In  a  similar  manner,  AR  object  recognition  can  identify  people, 
and  in  the  near  future  may  be  able  to  link  this  information  to  online  profiles  and  social  media 
data. 

Beyond  object  identification,  another  central  task  of  AR  systems  is  to  track  the  precise 
location  of  their  users  within  an  environment.  This  user  location  data  provides  the  AR  system’s 
virtual  mapping  application  a  reference  point  from  which  to  present  virtual  information  back  to 
the  user.  The  most  advanced  AR  systems  also  precisely  measure  the  head  location  of  their  users 
as  well  as  the  direction  of  their  visual  gaze.  With  this  data,  AR  systems  can  render  3D  virtual 
information,  anchored  to  a  user’s  perspective  within  a  given  environment.  For  instance,  this  is 
how  Microsoft’s  HoloLens  can  allow  users  to  attach  virtual  content  frames,  such  as  virtual 
televisions,  virtual  news  feeds,  and  virtual  teleconferences,  to  real-world  surfaces  and  then  move 

93  Lance  Eliot,  “Retail’s  New  Reality:  Invisible  Shopping  Centers  and  Virtual  Assistants,”  (CNBC,  April  24,  2015, 
http://www.cnbc .com/20 1 5/04/24/,  accessed  October  17,  2015). 
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about  in  space  and  retain  a  realistic  experience  due  to  corresponding  updates  in  the  geo-spatial 
perspectives  of  these  digital  frames  of  content.  These  digital  frames  appear  to  remain  in  place  as 
a  HoloLens  user  moves  about.  They  see  virtual  televisions  on  walls  when  in  reality  nothing  is 
there.  To  perform  the  complex  tasks  of  cueing  digital  content  and  rendering  it  appropriately,  a 
number  of  subcomponents  comprise  an  AR  system.  The  following  section  offers  a  generic  model 
of  these  subcomponents  and  how  they  interact  that  captures  the  essence  of  most  AR  systems. 


Figure  1  Generic  Augmented  Reality  System  Architecture 

Types  of  Sensors 

The  entry  points  for  data  into  an  augmented  reality  system  are  sensors  resident  in  the  AR 
display  device  or  relayed  to  it.  Similar  to  the  developments  from  sensor-fusion  in  smartphones, 
AR  display  devices  are  also  experiencing  a  revolution  in  new  applications  meant  to  leverage 
their  built-in  capabilities.  For  AR  systems,  however,  digitized  sensor  data’s  most  important 
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function  is  to  provide  coordinate  information  about  users,  their  environments,  and  objects  in 
space.  This  information  is  used  to  help  capture  information  about  a  user’s  environment  and  to 
later  help  render  virtual  content  back  into  a  user’s  FOV  on  their  AR  display  device.  Sensors 
collect  various  forms  of  analog  data  from  an  environment  and  then  digitize  it.  Once  this  analog 
data  is  collected  and  digitized,  it  can  be  combined  across  sensor  types  in  a  process  known  as 
sensor-fusion.  This  fusion  of  information  is  essential  for  high-quality  AR  experiences.94  Today, 
AR  systems  use  a  wide  variety  of  sensors  to  include  the  following. 

Optical  Cameras.  Digital  optical  cameras  use  an  electronic  sensor,  typically  a 
complementary  metal-oxide  semiconductor  (CMOS)  or  charge  coupled  device  (CCD)  to  capture 
images.  The  lens  on  an  optical  camera  focuses  the  light  onto  the  electronic  sensor.  The  optical 
properties  of  the  lenses  change  the  focus  of  the  camera.  The  size  of  its  aperture  determines  the 
amount  of  light  that  can  enter  the  lens.  Most  AR  systems  rely  on  at  least  one  optical  camera. 

Depth  Camera.  Normally,  depth-sensing  cameras  would  require  multiple  visual  inputs. 
However,  new  single  lens  depth  cameras  combine  new  LEDs  arrays  and  machine  learning  to 
register  depth.  These  new  depth-sensing  cameras  have  been  produced  largely  to  capture  user’s 
hand  motions  and  gestures  as  inputs  to  control  AR  interactions  with  virtual  content. 

Infrared  Camera.  Infrared  cameras  use  an  electronic  sensor  to  capture  infrared  radiation. 
Infrared  radiation  is  radiant  electromagnetic  energy  invisible  to  the  human  eye  with  a  frequency 
range  from  700  nanometers  to  1  mm.  Infrared  cameras  offer  two  unique  applications  for  AR 
systems.  First,  they  provide  another  robust  means  of  capturing  data  from  an  environment  to  build 
a  virtual  map  of  one’s  surroundings.  When  paired  with  infrared  markers,  they  can  be  used  to 
identify  marked  objects  invisible  to  the  human  visual  system.  In  this  way,  AR  developers  that 

94  J.D.  Hoi,  T.B.  Sch'on,  F.  Gustafsson,  and  P.J.  Slycke,  “Sensor  fusion  for  augmented 
Reality,  ”  (In  International  Conference  on  Information  Fusion,  Florence,  Italy,  August  2006). 
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craft  AR  experiences  can  declutter  a  visual  environment,  and  still  provide  their  systems  easily 
identifiable  markers  that  do  not  require  robust  ambiguous  object  discrimination  capabilities. 

Accelerometers  measure  g-force  acceleration.  They  typically  are  employed  in  motion 
sensing  applications  to  continuously  calculate  changes  in  one’s  position.  Beyond  navigation 
applications  they  have  also  been  incorporated  in  biometric  applications  that  seek  to  measure  the 
body-movement  of  users.  In  addition,  smartphones  and  mobile  devices  use  them  to  indicate 
changes  in  screen  orientation  so  they  can  trigger  portrait  and  landscape  views. 

Magnetometers  measures  the  magnetization  of  material  and  in  some  cases  the  direction  of 
the  magnetic  field  at  a  point  in  space.  In  mobile  devices,  and  AR  systems,  magnetometers 
typically  serve  as  compasses  along  three-axis.  They  have  also  been  used  to  track  changes  in  the 
magnetic  field  of  users.  This  information  can  be  used  to  identify  gestures  and  controls. 

Solid-State  Compasses.  A  solid-state  compass  determines  direction  relative  to  the  earth 
surface  for  navigational  purposes.  These  compasses  rely  on  two  or  three  magnetometers  and  a 
microprocessor  to  calculate  heading. 

Global  Positioning  System  (GPS)  Receiver.  GPS  receivers  collect  cell-tower  and  space- 
based  navigational  information  about  one’s  location,  as  well  as  timing  cues.  Cell-tower  GPS 
receivers  determine  location  based  on  signal  measurement  strength,  and  timing  to  receive  GPS 
signals  from  surrounding  towers  and  then  calculate  a  user’s  location  based  on  triangulation 
methods.  Space-based  GPS  receivers  use  radio  waves  from  space,  using  the  same  general 
methods  as  cell-tower  GPS  receivers,  to  determine  a  user’s  location. 

Gyroscopes.  Mobile  devices  and  AR  systems  use  gyroscopes  to  calculate  orientation  and 
rotation.  They  have  been  incorporated  to  measure  movement  within  3D  space.  When  combined 
with  accelerometers,  they  can  accurately  calculate  changes  along  3-axis. 
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Barometers.  In  mobile  devices  barometers  help  GPS  receivers  to  get  a  faster  lock  by 
providing  altitude  data  instantaneously. 

Pressure  Sensors.  Pressure  sensors  measure  the  ambient  air  pressure.  These 
measurements  can  be  used  to  measure  altitude  by  using  the  pressure  difference  between  two 
locations.  These  type  of  sensors  are  typically  used  in  touchscreen  technologies. 

Proximity  Sensors.  Proximity  sensors  in  mobile  devices  can  detect  when  a  phone  is 
brought  to  a  user’s  face  during  a  call.  Most  mobile  devices  use  infrared  proximity  sensors  so  they 
can  reduce  display  power  use  by  turning  off  the  LCD  and  disabling  the  touch-screen  during 
phone  calls. 

Radio-Frequency  Identification  (RFID).  RFID  uses  wireless  electromagnetic  fields  to 
transmit  and  receive  data.  This  technology  is  typically  used  to  automatically  identify  and  track 
tags  or  markers  attached  to  objects. 

Bluetooth.  Another  wireless  technology  used  to  transmit  and  receive  data  over  short 
distances  is  Bluetooth.  Bluetooth  uses  UHF  radio  waves  in  the  ISM  band  that  ranges  from  2.4  to 
2.485  GHz.  Many  AR  systems  use  Bluetooth  to  transfer  data  to  AR  display  devices  from  a 
separate  AR  component  that  houses  the  central  augmented  reality  processor  and  various  AR 
modules. 

Wi-Fi.  Wi-Fi  provides  a  local  area  wireless  networking  capability.  Data  is  transferred 
typically  at  the  2.4  GHz  and  5  GHz  frequency  radio  bands.  Similar  to  Bluetooth,  AR  utilizes  Wi¬ 
Fi  connections  to  communicate  data  to  and  from  AR  systems  holistically  and  to  particular 
wireless  AR  display  devices. 
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Microphone.  Microphones  capture  acoustic  data  and  transform  it  to  a  digital  format 
through  an  acoustic-to-electric  transducer  or  sensor.  In  AR  systems,  built-in  microphones  can 
register  a  user’s  commands  as  well  as  other  sounds  from  a  user’s  environments. 

Capturing  and  Tracking  Modules 

AR  capturing  modules  combine  sensor  information  with  geo-locational  3D  coordinate 
information  to  provide  geo-registered  data  about  a  user’s  environment.  As  sensors  digitize  an  AR 
user’s  environment,  they  identify  markers  that  will  be  referenced  later  to  anchor  virtual 
information.  Capturing  modules  use  sensor-specific  reference  points  to  link  this  anchoring 
information  to  raw  sensor  data.  This  information  is  then  used  to  render  virtual  information. 

Tracking  modules  use  the  anchoring  information  provided  by  the  capturing  module  as  a 
means  to  track  the  locations  of  a  user’s  augmented  reality  device  and  objects  from  the  real 
environment.  AR  systems  use  tracking  modules  to  add  virtual  objects  into  a  user’s  perception  of 
the  real-world.  The  geo-location  of  the  AR  device  and  the  markers  created  by  the  capturing 
module  serve  as  the  basic  elements  used  to  orient  information  within  the  virtual  environment.  As 
AR  users  move  throughout  an  environment,  the  tracking  module  continuously  reregisters  their 
position  and  the  relative  position  of  objects  in  both  real  and  virtual  environments.  This  allows 
AR  systems  to  fix  the  location  of  virtual  objects  and  keep  them  oriented  correctly  in  response  to 
an  AR  user’s  real-time  movements  in  space. 

Currently,  AR  systems  vary  in  how  they  capture  and  track  reference  points  and  anchor 
digital  content.  Some  rely  on  simple  pattern  recognition  of  basic  shapes  and  markers.  In  these 
systems,  when  a  pattern  is  recognized  an  AR  system  will  replace  that  area  with  virtual 
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information.95  For  example,  a  wall  in  a  room  may  be  registered  as  a  reference  point  and  used  to 
anchor  a  virtual  monitor  on  it.  When  an  AR  user  looks  toward  the  blank  wall  while  wearing  an 
AR  display  device  the  virtual  monitor  will  appear. 

Another  method  employed  by  AR  systems  to  capture  and  anchor  information  is  the 
outline  method.  This  technique  recognizes  different  parts  of  an  AR  user’s  body,  such  as  the 
hands  or  face,  and  then  tracks  the  movements  of  these  parts  so  AR  users  can  interact  with  virtual 
objects  using  natural  movements.96  For  instance,  when  an  AR  user  reaches  for  a  virtual  object 
and  makes  a  grasping  motion,  this  action  can  trigger  a  protocol  that  can  move  and  rotate  the 
virtual  object  in  3D  space.  In  this  manner,  AR  users  can  interact  with  virtual  objects. 
Furthermore,  simple  hand  gestures  may  be  used  in  the  way  that  swipes,  pinches,  and  taps  control 
smartphones  and  tablets  today  -  though  instead  of  touching  a  physical  surface  these  motions 
could  take  place  in  space.  Microsoft  has  already  developed  these  types  of  controls  for  their 
Kinetic  system  and  they  will  likely  be  employed  in  HoloLens. 

The  location  method  utilizes  detailed  GPS  or  triangulation  methods  to  precisely  overlay 
and  orient  digital  content  relative  to  an  AR  user’s  location.  When  discrete  shapes  and  surfaces 
are  not  readily  available,  such  as  in  natural  landscapes,  simply  knowing  the  precise  locations  of 
objects  through  GPS  sensors,  can  allow  for  the  tracking  and  display  of  virtual  information.  For 
military  forces  accustomed  to  monitoring  blue  forces  as  icons  on  2D  screens,  this  information 
can  now  be  projected  in  AR  displays  in  3D  real-world  environments.  An  operator  on  patrol  can 
look  over  a  valley  and  see  virtual  icons,  displayed  to  represent  depth  and  distance,  of  blue  forces 
in  space.  The  same  is  true  for  marked  targets  and  known  areas  of  interest.  They  can  all  have 
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virtual  content  associated  with  them  so  that  when  the  time  comes  when  an  AR  user  comes  within 


range  that  enhancing  information  can  appear  overlaid  on  that  location. 

Finally,  the  surface  method  uses  surfaces,  walls,  and  screens  that  respond  to  touch  to 
display  virtual  content.  As  users  touch  and  interact  these  two  dimensional  surfaces  they  can 

no 

manipulate  and  interact  with  virtual  objects.  ARES  has  created  an  AR  sand  table  whereby  users 
can  interact  with  virtually  displayed  earth  and  objects  and  manipulate  their  locations  through 
hand  gestures  that  correspond  to  how  someone  would  naturally  build  a  sand  table  today."  The 
acts  of  pushing,  scooping,  pulling  virtual  content  immediately  renders  changes  in  these  virtual 
images  as  if  they  were  real  objects. 

Some,  but  not  all,  AR  systems  combine  aspects  from  each  of  these  four  types  of 
capturing  models.  The  more  advanced  systems  track  user’s  actions,  and  captured  detailed 
information  about  environments,  so  they  can  render  virtual  content  in  a  manner  that  best 
deceives  the  human  sensory/perceptual  system  from  distinguishing  the  virtual  from  the  real.  As 
sensors  decrease  in  cost,  and  increase  in  capacity,  the  accuracy  of  all  AR  capturing  modules  will 
increase.  At  the  same  time  as  the  power  of  capturing  models  increases  and  their  ability  to 
identify  and  recognize  objects  expands  -  the  catalogue  of  virtual  AR  experiences  will  only 
expand  and  become  more  powerful.100 
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Virtual  Information 


Virtual  information  comes  from  augmented  reality  applications  and  is  then  rendered  in 
the  AR  display  device.  Multiple  AR  applications  may  run  at  the  same  time  providing  unique 
content  in  a  hybrid  manner.  This  information  can  be  rendered  in  either  two  or  three  dimensions 
using  one  of  the  four  capturing  methods  previously  described  (pattern,  outline,  location,  or 
surface).  The  majority  of  all  advertised  AR  systems  can  render  2D  enhancing  information.  This 
is  sometimes  described  as  mediated  reality.  The  content  is  not  necessarily  anchored  to  a  person’s 
perspective.  If  they  move  their  head  the  AR  data  remains  fixed  in  their  FOV.  This  does  not  mean 
the  data  itself  is  not  tailored  to  the  environment  or  the  user’s  experience  in  it. 

The  more  complex  AR  virtual  information,  and  the  data  that  most  would  define  as  true 
augmented  reality,  is  that  which  appears  overlaid  and  anchored  to  discrete  locations  within  a 
user’s  real-world  environment.  For  example,  AR  users  may  experience  3D  virtual-telepresence 
whereby  an  offsite  co-worker  appears  seated  at  a  conference  table  in  the  same  room  as  the  AR 
user.  They  may  also  see  3D  rendered  blue  prints  and  schematics  they  designed  projected  in  real 
space.  They  can  move  around  these  virtual  objects  and  examine  them  from  new  angles.  On  the 
future  site  of  a  building,  an  architect  may  see  a  virtual  building  and  have  the  capability  to  add  or 
remove  structures  in  the  order  in  which  they  will  be  constructed.  In  the  military,  particularly  in 
the  Navy,  having  this  type  of  capability  would  be  of  great  benefit.  Virtual  information  about  new 
additions  to  a  vessel  could  be  projected  inside  a  vessel  for  engineers  to  better  visualize  how  they 
would  appear  and  fit  within  existing  structures. 

Augmented  reality  devices  may  have  multiple  software  applications  running  and 
displayed  for  users.  The  information  used  and  presented  by  these  various  AR  applications  may 
come  from  various  sources.  Aside  from  the  AR  content  tailored  to  the  specific  AR  system,  future 
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AR  systems  will  likely  incorporate  other  third-party  AR  applications.  Currently,  Microsoft’s 
HoloLens  will  run  Windows  10,  and  as  such  it  will  be  able  to  load  and  run  any  application 
compatible  with  this  operating  system.  Soon,  AR  users  may  open  their  email  program,  cable 
news  feed,  social  media  application  and  attach  this  virtual  content  to  various  walls  and  surfaces 
in  their  office  and  leave  them  running  as  they  move  in  and  out  of  that  space. 

Virtual  information  may  originate  from  processing  at  the  AR  display-device  or  it  may 
come  from  content  servers  accessed  through  a  mobile  internet  connection.  By  originating  at  the 
AR  display  device  the  latency  to  present  rendered  information  decreases;  thereby,  increasing  the 
responsiveness  of  these  AR  systems.  However,  as  AR  services  begin  to  enter  commercial 
markets,  more  digital  AR  content  and  out-sourced  processing  may  reside  away  from  the  AR 
display  device.  Information  from  external  data  sources  such  as  Wiki-like  content  servers  and 
social  media  sites  will  inevitably  have  to  be  stored  elsewhere  and  accessed  dynamically.  The 
characteristics  of  how  AR  virtual  information  is  presented  are  determined  by  the  AR  rendering 
module. 

Rendering  Module 

Rendering  modules  combine  information  from  a  real  environment  with  virtual 
information.  They  operate  within  and  are  controlled  by  an  AR  system’s  Centralized  Augmented 
Reality  Processor  (CARP).  Rendering  modules  take  an  AR  application’s  virtual  infonnation  and 
prepare  it  to  be  presented  back  to  the  user.  They  utilize  what  can  be  thought  of  as  a  virtual 
camera,  identical  to  the  cameras  found  at  the  AR  display  device,  to  prepare  virtual  objects  to  be 
displayed  in  a  convincing  manner.  They  use  the  geo-locational  3D  coordinate  information 
provided  by  the  tracking  module  to  place  infonnation  in  a  virtual  3D  rendition  of  a  user’s  real 
environment.  Once  placed,  the  rendering  module  takes  the  2D  and  3D  virtual  information 
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injected  by  the  AR  applications  and  makes  it  appear  at  the  proper  locations,  in  the  correct  size, 
shape,  and  orientation  for  the  user’s  AR  device  display. 

Centralized  Augmented  Reality  Processor  (CARP) 

The  CARP  serves  as  the  primary  control  mechanism  within  AR  systems.  Much  like  a 
Central  Processing  Unit  (CPU)  controls  the  overall  actions  of  computers  the  CARP  coordinates 
the  actions  of  AR  systems.  It  is  the  digital  brain  of  AR  systems.  It  transmits  sensor  inputs  and 
manages  processing  resources  shared  by  the  capturing,  tracking,  and  rendering  modules.  It  also 
provides  processing  and  memory  resources  for  any  AR  application  running  at  the  AR  device 
while  also  receiving  corresponding  virtual  information  from  external  AR  applications  via  its 
mobile  internetwork  interface.  Finally,  it  prioritizes  and  organizes  virtual  information  to  be 
rendered  and  projected  by  the  AR  display-device. 

Display  Devices 

As  previously  noted,  a  number  of  different  AR  display-devices  exist.  These  display- 
devices  come  in  many  form  factors  and  will  continue  to  evolve  to  become  lighter  and  less 
obtrusive  as  their  sub-components  decrease  in  size  and  increase  in  capability.  The  earliest  forms 
were  HUDs  and  HMDs.  Both  of  these  can  either  use  optical  see-through  or  video  see-through. 
The  growth  of  smartphone  and  tablets  generated  AR  applications  that  simply  displayed  AR 
content  using  their  displays.  In  a  similar  manner  TV  displays  and  kiosks  can  present  AR 
rendered  virtual  information.  As  previously  outlined,  much  of  the  current  excitement  about  the 
future  of  AR  systems,  however,  has  arisen  out  of  the  growing  capability  of  AR  glasses  and 
contact  lenses.  As  these  devices  decrease  in  size  and  become  less  conspicuous  they  will  attract 
more  users. 
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Augmented  Reality  and  Enhanced  Cognition 


The  previous  sections  discussed  the  development  AR  technologies  and  briefly  described 
some  of  their  current  capabilities.  The  following  section  will  explore  how  AR  systems  will  affect 
human  cognition.  As  an  information  technology,  AR  systems  affect  how  their  users  gain 
information  and  understand  it.  What  makes  AR  revolutionary  is  that  it  will  also  redefine  how  its 
users  explore  and  interact  with  their  environments.  The  use  of  AR  systems  affects  basic  elements 
of  cognition,  such  as  how  attention  is  directed  and  maintained.  In  addition,  they  also  affect  more 
complex  cognitive  activities  such  the  integration  of  new  information  into  mental  representations 
and  frameworks. 

Human  thought  can  be  understood  as  a  continuous  perception-action  cycle.  This  cycle 
consists  of  flows  of  sensory  signals  through  specialized  regions  of  the  brain  organized  to  process 
discrete  forms  of  information.  This  processing  identifies  patterns  in  order  to  develop  predictive 
cuing  and  these  flows  of  information  develop  circuits  from  the  senses  to  higher-level  cognitive 
processes  that  guide  goal-directed  behavior.101  Signals  from  sensory  organs  follow  discrete 
paths,  both  to  higher  level  processing  centers  as  well  as  to  primitive,  reflexive  mental  nodes. 
Primitive  mental  functions,  which  require  little  goal-directed  control,  allow  humans  to  respond 
automatically  to  various  stimuli.  Higher  level  mental  functions,  process  and  organize  incoming 
information  and  distribute  it  to  other  mental  nodes  for  processing.  These  higher  level  mental 
functions  do  not  only  receive  signal  information  for  processing  but  they  also  direct  lower  level 
sensory/perceptual  actions.  They  do  this  through  the  control  of  a  central  executive  that  allocates 


101  Vassilis  Cutsuridis,  Amir  Hussain  and  John  G.  Taylor  (eds):  “Perception-action  Cycle.  Models,  Architectures, 
and  Hardware,”  (Cognitive  Neurodynamics,  Vol.  5,  Issue  4,  November,  2011),  pp.  41 1-412. 
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attention  and  working  memory.  “  This  is  how  humans  can  quickly  move  from  watching  a 
football  game,  to  listening  to  and  encoding  details  about  a  story  someone  is  telling,  and  then 
return  back  to  the  game  and  still  recall  both  the  progress  on  the  football  field  as  well  as  details 
from  the  narrative. 

The  central  executive  is  a  top-down  cognitive  mechanism.  The  sensory/perceptual  feeds 
to  the  brain  are  bottom-up  cognitive  mechanisms.  Both  sub-systems  have  limits  and 
constraints.  Many  often  complain  that  they  wished  they  had  better  memories,  or  that  they 
could  better  focus.  AR  systems  may  be  able  to  assist  or  impede  both  functions  depending  on 
their  design.  Understanding  not  only  how  attention  is  captured  but  also  how  it  is  controlled  is 
central  to  understanding  precisely  how  AR  devices  impact  both  top-down  as  well  as  bottom-up 
cognitive  processing. 

Augmented  Reality  and  Sensory/Perceptual  Experiences 

By  their  nature,  AR  devices  capture  and  redirect  AR  users’  attention.  Attention  serves  as 
both  a  “filter”  for  bottom-up  processing,  restricting  noise  and  registering  signals,  and  a  “fuel”  for 
top-down  processing  providing  mental  focus.104  Particularly  salient  stimuli  in  an  environment 
can  automatically  capture  one’s  attention.  The  idiosyncratic  needs  of  an  organism,  combined 
with  its  particular  sense  organs,  determine  what  is  salient  and  what  is  not.105  The  smell  of  a 
female  dog  in  heat  may  not  register  for  most  human  pet  owners,  but  it  sure  attracts  the  attention 
of  their  dogs. 


102  A.D.  Baddeley  &  G.  Hitch,  “Working  Memory,”  (In  G.H.  Bower  (Ed.),  The  Psychology  of  Learning  and 
Motivation:  Advances  in  Research  and  Theory,  Vol.  8,  Academy  Press,  NY),  pp.  47-89. 

103  George  A.  Miller,  "The  Magical  Number  Seven,  Plus  or  Minus  Two:  Some  Limits  on  Our  Capacity  for 
Processing  Information,"  (Psychological  Review,  63,  (2),  1956),  pp.  81-97. 

104  Christopher  D.  Wickens,  “Attention,”  (In  John  D.  Lee  &  Alex  Kirlik  (Eds.)  The  Oxford  Handbook  of  Cognitive 
Engineering,  2013),  pp.  37-56. 

105  David  Oderberg,  Perceptual  Relativism,  (Philosophia,  16,  T9,  1986). 


39 


Of  the  five  primary  senses  in  humans,  the  visual  system  tends  to  dominate  sensory 
experiences  and  the  direction  of  attention.  A  flash  of  light,  sudden  appearance  of  an  object,  or 
rapid  movement  automatically  captures  human  visual  attention  and  causes  one’s  gaze  to  fixate 
on  the  source  of  the  salient  events.106  In  an  earlier  time  in  humankind’s  history,  these  events  may 
have  indicated  the  presence  of  a  predator  to  be  avoided  or  an  opportunity  during  a  hunt.  Today, 
the  human  visual  system  is  still  tuned  to  these  types  of  salient  cues.  Augmented  reality  display 
devices  naturally  capture  the  attention  of  their  users  because  they  introduce  these  types  of  onsets 
and  cues.  Therefore,  the  first  very  basic  way  in  which  AR  systems  will  change  how  their  users 
experience  their  worlds  is  by  automatically  redirecting  their  visual  attention. 

Beyond  their  visual  displays  many  AR  systems  also  include  auditory  outputs.  Some  of 
the  basic  auditory  outputs  simply  provide  for  2D  sound  experiences.  Users  can  hear  sounds  and 
language  either  in  mono  or  stereo  formats.  In  more  advanced  AR  systems,  auditory  information 
is  presented  in  a  manner  that  gives  the  illusion  of  coming  from  3D  space.  These  outputs  are 
designed  to  give  the  illusion  of  sounds  coming  naturally  from  one’s  environment.  Combining 
3D  sounds  with  dynamic  visual  AR  content  creates  a  multi-modal  sensory  experience  that  further 
increases  the  verisimilitude  of  the  AR  experience  for  users. 

In  addition,  AR  wearables  exist  that  interact  with  a  user’s  haptic  sensory/perceptual 

1  AO 

experiences.  They  vibrate  and  pulse  based  on  external  cues.  In  this  manner,  they  can  give  the 
illusion  of  physical  contact  and  interaction  with  virtual  objects.  For  example,  researchers  at 
Disney  created  an  AR  tactile  technology  that  generates  the  same  haptic  experience  of  feeling  real 

106  Ran  Carmi  and  Laurent  Itti,  “Visual  Causees  Versus  Correlates  of  Attentional  Selection  in  Dynamic  Scenes,” 
(Vision  Research,  46,  2006),  pp.  4333-4345. 

107  William  Shankin,  “Microsoft  HoloLens:  An  AR  Headset  That  Makes  You  see  3D  Holograms  in  Your  Living 
Room,”  (Gizmag,  January  21,  2015,  http://www.gizmag.com/microsoft-hololens-details/35708/,  accessed  May  15, 
2015). 

108  Olivier  Bau  and  Ivan  Poupyrev,  “REVEL:  Tactile  Feedback  Technology  for  Augmented  Reality,  "  (ACM 
Transactions  on  Graphics,  Vol.  31,  No.  4,  Article  89,  2012). 
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objects  by  creating  an  oscillating  electrical  field  around  a  user’s  fingers.  When  users  reached  for, 
and  interacted  with  virtual  objects,  they  reported  experiencing  the  same  type  of  haptic  sensations 
as  when  they  interacted  with  real  items  and  materials.109 

Beyond  affecting  these  basic,  bottom-up  sensory/perceptual  experiences,  AR  systems 
also  affect  other  fundamental  forms  of  cognition  such  as  spatial  perception  and  reasoning.  When 
humans  navigate,  they  rarely  have  to  devote  a  lot  of  top-down  attention.  Various  bottom- up 
cognitive  faculties  allow  humans  to  navigate  to  and  from  locations  and  remain  oriented  with  an 
environment.  The  use  of  AR  systems  provides  additional  spatial  cues  and  navigational  aids  to 
assist  AR  users  to  know  their  location  and  maintain  their  desired  direction  of  travel.  At  times 
when  the  human  spatial  perception  system  struggles,  such  as  when  visibility  is  limited,  or  in 
environments  where  reliable  spatial  cues  are  occluded  or  obstructed,  AR  systems  can  provide  the 
necessary  means  to  remain  oriented.  AR  systems  can  achieve  this  by  displaying  simple 
directional  arrows  or  geo -registering  a  target  destination  and  displaying  it  in  a  user’s  FOV. 

In  summary,  it  is  clear  that  the  use  of  AR  systems  impacts  human  sensory/perceptual 
experiences  by  tricking  the  human  mind  into  believing  that  virtual  content  is  real.  Augmented 
reality  systems  can  present  realistic  signals,  primarily  through  the  visual  and  auditory  systems, 
but  also  in  new  haptic  apparatuses.  In  so  doing,  AR  systems  can  capture  and  redirect  the 
attention  of  their  users.  This  bottom-up  signaling  can  thus  force  users  to  focus  on  virtual 
information  over  the  real-world.  The  challenge  for  AR  system  designers  is  to  recognize  just  how 
distracting  these  devices  can  be.  They  must  put  in  place  appropriate  safety  measures  to  ensure 
users  do  not  pay  more  attention  to  digital  signals  than  real-world  dangers. 


109 
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Augmented  Reality  and  Information  Behavior 


Beyond  affecting  bottom-up  sensory/perceptual  experiences  AR  will  also  have  a 
profound  effect  on  top-down  cognitive  processes  -  in  particular  those  that  comprise  the  behavior 
employed  in  learning.  AR  systems  will  transform  the  way  in  which  people  interpret  and  explore 
their  environments.  They  will  alter  how  they  seek  out  information  and  use  it.  By  employing  AR 
systems  throughout  their  daily  lives,  users  will  grow  accustomed  to  the  many  ways  in  which  AR 
will  change  their  information  behavior.  To  explain  the  catalogue  of  these  effects  the  following 
sections  will  use  David  Ellis’s  descriptions  of  key  human  information  behaviors. 

Starting  when  people  begin  to  look  for  new  information  they  employ  predictable  patterns 
of  information  seeking  behavior.  In  general,  people  begin  by  looking  for  sources  of  expertise  (i.e. 
a  subject  matter  expert,  relevant  books  or  publications,  or  online  search. 1 10  The  relative 
background  knowledge  of  the  individual  looking  to  learn  combined  with  their  particular  learning 
style  affects  their  approach.  However,  most  begin  by  looking  for  a  credible  reference.  After  the 
birth  of  the  internet,  sources  of  expertise  shifted.  With  the  expansion  of  AR  systems,  connected 
not  only  to  the  internet,  but  also  cloud  processing  and  virtual  personal  assistants,  another  shift  in 
credible  references  will  occur.  More  people  will  turn  to  Facebook’s  M,  Apple’s  Siri,  Microsoft’s 
Cortana,  and  Google  Now  for  answers  before  talking  to  a  real  person  or  visiting  a  library.  What 
AR  systems  provide  is  continuous  access  to  these  information  sources  and  a  new  symbiosis 
between  the  question  asker  and  the  answer  giver. 

Those  wearing  AR  devices  will  gain  the  ability  to  share  their  AR  sensor  data  with  semi¬ 
artificial  intelligent  services  -  thereby  letting  the  Siris  of  the  future  to  not  only  hear  voice 
commands  but  also  “see”  an  AR  user’s  digitized  surroundings.  From  this,  dramatic  changes  in 

1 10  David  Ellis,  The  Derivation  of  a  Behavioral  Model  for  Information  Retrieval  System  Design ,  (Doctoral  Thesis 
submitted  at  University  of  Sheffield,  August,  1987). 
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how  humans  start  to  look  for  information  will  ensue.  In  a  not  too  distant  future,  these  services 


may  begin  to  anticipate  the  questions  their  users  will  generate  to  an  even  greater  extent  than  they 
already  can.1 1 1  They  may  use  geo-referenced  data  to  offer  up  insights  and  recommendations 
based  on  one’s  location,  time  of  day,  or  registered  biometric  state.  For  example,  when  an  AR 
user’s  system  registers  a  drop  in  blood  sugar  at  a  time  when  they  are  near  a  series  of  restaurants, 
it  may  automatically  offer  suggestions  for  recommended  restaurants  it  predicts  a  user  may  ask 
about.  Or,  if  an  AR  system  registers  a  rare  word,  or  unusual  term  used  by  someone  addressing  an 
AR  user,  it  may  display  a  definition  in  their  FOV  to  reference.  Already  today,  real-time 
translation  software  is  present  in  Google  Glass,  thereby  allowing  it  to  translate  foreign 
conversations  for  its  users.  Currently,  AR  systems  do  not  have  any  direct  links  to  such  semi¬ 
artificial  intelligent  services,  but  many  can  connect  to  the  internet  both  receiving  and  sending 
information.  This  two-way  data  flow  will  increase  learning  for  both  AR  systems  and  AR  services 
looking  to  understand  the  behavior  of  their  users. 

Chaining.  The  next  action  most  people  take  as  they  begin  to  seek  out  information  is  to 
link  new  information  with  the  old.  '  This  is  what  is  known  as  chaining.  As  people  learn  more 
about  a  given  subject  they  often  look  for  other  source  materials  used  to  frame  the  information  or 
support  a  claim.  They  compare  new  information  with  what  they  already  know  and  determine  if  it 
fits  in  their  current  understanding,  or  somehow  alters  it.  ~  AR  systems  can  facilitate  this  process. 
AR  systems  allow  users  to  gain  enhancing,  explanatory  information  in  real-time.  This  differs 
from  smartphones  and  computers  that  require  users  to  shift  their  attention  away  from  their 
current  environment  or  perspective  in  order  to  interact  with  various  search  engines. 

111  PRWeb,  “Deep  Learning  and  Digital  Companions  are  the  New  Virtual  Assistants,”  (Virtual  Strategy  Magazine, 
October  24,  2015,  http://www.virtual-strategv.com/2015/10/24/deep-learning-and-digital-companions-are-new- 
virtual-assistants#axzz3pUT0RmRL,  accessed  October  24,  2015). 

112  Ellis,  1987. 
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AR  systems  enable  users  to  better  understand  and  visualize  the  relationships  between 
various  forms  of  information  by  helping  them  to  rapidly  access  relevant  explanatory  information. 
For  example,  AR  systems  have  been  employed  in  several  maintenance  tasks.  For  example,  BMW 
has  developed  an  AR  system  its  mechanics  can  where  that  displays  virtual  information  about  its 
cars’  engines.114  Mechanics  can  use  AR  systems  to  not  only  see  virtual  configurations  of  systems 
so  they  can  better  identify  unknown  parts,  but  they  can  rapidly  retrieve  additional  information 
about  these  unknown  components  through  a  voice  directed  search  engine.  They  no  longer  have 
to  look  away  from  the  task  at  hand  while  seeking  to  expand  their  understanding  or  solve  a 
particular  problem.  At  the  same  time,  if  they  wish  to  pause  what  they  are  doing,  and  learn  more 
about  other  factors  related  to  a  given  problem,  they  can  rapidly  access  further  enhancing 
information.  In  this  manner,  they  are  able  to  chain  the  information  they  discover. 

Searching.  Browsing  concerns  information  behavior  involving  semi-directed  or  semi- 
structured  searches  for  information.115  Though  inefficient  at  times,  browsing  allows  people  to 
quickly  scan  a  list  of  contents  and  select  what  items  may  be  of  interest.116  To  date,  RSS  feeds  and 
the  ability  to  “follow”  a  person  or  organization  allow  users  to  automatically  receive  collections 
of  information  on  topic  areas  in  which  someone  is  particularly  interested.  In  the  future,  AR 
systems  will  reduce  the  time  to  find  relevant  information  by  learning  what  browsing  patterns  its 
users  employ  and  what  particularly  content  is  of  most  interest  to  them.  With  this  data,  AR  can 
begin  to  predict  what  information  a  user  may  need  or  want  in  a  given  environment  or  when  faced 
with  a  particular  type  of  problem. 


4  Victoria  Woollaston,  “End  of  the  Mechanic?  BMW  Smart  Glasses  Make  it  Possible  for  ANYONE  to  Spot  and 
Fix  a  Car  Engine  Fault  Just  by  Looking  at  it,”  (Daily  Mail,  January  21,  2014, 

http://www.dailymail.co.uk/sciencetech/article-2543395/The-end-mechanic-Smart-glasses-make-possible-fix-car- 

engine-iust-looking-it.html,  accessed  October  17,  2015). 

115  Ellis,  1987. 

116  Ibid. 
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AR  systems  can  first  achieve  this  through  geo-fencing  techniques  that  cue  virtual 
information.  Geo-fencing  involves  pairing  digital  information  and  content  with  real-world, 
geographical  boundaries.  Geo-fencing  applications  have  been  designed  for  various  entertainment 
and  tour  services.  For  example,  when  customers  visit  various  wineries  they  automatically  receive 
alerts  and  details  about  the  wineries  wines  via  their  mobile  devices.  A  number  of  commercial 
retailers  are  looking  to  employ  similar  geo-fencing  techniques  so  that  potential  customers  only 
get  notifications  when  they  are  within  range  of  one  of  their  products  in  a  store.  Through  AR 
systems,  geo-fencing  would  not  only  affect  how  people  browse  for  information  in  digital  spaces 
but  also  how  they  interact  with  objects  in  their  real-world  environments. 

Another  means  by  which  AR  can  affect  browsing  is  by  monitoring  where  people  direct 
their  attention.  A  number  of  AR  display  devices  tracking  head  and  eye  movements  so  they  can 
render  virtual  content  in  the  correct  position  and  orientation  relative  to  a  user’s  FOV.  This  same 
data,  however,  can  be  used  to  track  what  objects,  people,  and  locations  gain  a  user’s  attention. 
When  an  AR  system  has  this  data,  it  can  then  begin  to  automatically  search  for  enhancing 
information  and  present  it  if  desired  to  the  AR  user.  Through  this  type  of  tracking,  future  AR 
systems  will  affect  how  people  browse  and  search  their  environments. 

Differentiating.  As  people  seek  out  information  they  prefer  to  gain  new  knowledge  from 
trusted  sources.  Differentiating  is  the  information  behavior  concerned  with  the  credibility  and 
quality  of  a  given  source  of  information.  Future  AR  systems  will  impact  this  information 
behavior  in  a  couple  of  significant  ways.  First,  through  the  use  of  the  afore-mentioned  semi¬ 
artificial  intelligence  services  such  as  M,  Siri,  Cortana,  Google  Now,  etc.,  AR  users  may  drift 

117  Shel  Israel,  “How  VinTank  is  Building  a  Geo-Fence  Around  Wine  Country,”  (Forbes,  March  20,  2013, 
http://www.forbes.com/sites/shelisrael/2013/03/20/how-vintank-is-building-a-geo-fence-around-wine-country/, 

accessed  October  17,  2015. 
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away  from  trusting  human  expertise  to  relying  on  computer-aided  understanding  to  a  greater 
extent.  In  addition,  AR  systems  may  display  rating  and  review  indicators  of  people  that  users 
interact  with. 

To  the  extent  that  future  AR  systems  can  recognize  people  that  come  within  range  of  an 
AR  user’s  optical  cameras  and  then  look  up  their  level  of  expertise,  or  reputation  through  an 
online  database,  they  could  then  offer  some  form  of  indication  to  an  AR  user.  For  instance,  when 
speaking  with  an  electrician  an  AR  user  may  see  a  star  indicator,  such  as  four  stars  out  of  five, 
displayed  as  a  semi-transparent  halo  floating  over  their  head.  From  this  enhancing  information, 
the  AR  user  would  automatically  know  that  previous  clients  thought  highly  of  their  performance. 
Similar  to  what  online  customers  today  can  see  when  reviewing  potential  products  or  services, 
future  interactions  with  others  may  trigger  AR  systems  to  present  various  indicators  about  a 
person’s  expertise,  credibility,  reputation,  etc. 

Extracting.  As  people  learn  new  information  they  try  to  focus  on  what  is  important  to 
them  while  filtering  out  other  distracting  details.  Extracting  is  concerned  with  selectively 
drawing  only  the  information  most  relevant  to  a  user  at  a  given  time.119  Identifying  what  sources 
of  information  are  most  useful  is  a  pre-requisite  of  any  extraction  effort.  The  increase  in  semi¬ 
artificial  intelligent  personal  assistant  services  combined  with  AR’s  ability  to  learn  from  a  user’s 
continuous  interaction  with  various  sources  of  information  across  a  variety  of  environments  will 
allow  AR  systems  to  better  present  the  most  relevant  information  for  its  users.  As  AR  systems 
gain  access  to  object  recognition  applications  and  services  they  will  be  empowered  to  rapidly 
identify  and  categorize  items  in  one’s  surroundings  and  thereby  help  prioritize  what  should  be 
attended  to. 


119  Ellis,  1987. 
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The  impact  of  AR  use  on  how  users  extract  information  could  take  many  forms.  Some 
users  may  become  solely  reliant  on  their  AR  systems  to  feed  them  information.  Those  less 
inquisitive  may  be  satisfied  with  an  AR  system’s  default  search  results.  Other  AR  users  may 
demand  more  of  their  systems  and  will  spend  more  time  finding  ways  to  rapidly  learn  all  there  is 
to  know  about  a  particular  setting,  situation,  object,  or  person.  Some  AR  systems  may  be 
dominated  by  marketing  applications  and  consequently  inundate  their  users  with  focused 
advertisements.  Still  others  may  be  specialized  for  niche  professions  such  as  doctors  and 
scientists  -the  virtual  information  of  these  particular  systems  may  cost  a  great  detail  or  require 
restrictive  licensing  or  service  contracts.  Only  time  will  tell  precisely  how  AR  systems  determine 
what  information  to  extract  and  present  to  their  users.  Regardless,  these  systems  will  have  an 
impact  on  where  their  users  direct  their  attention  and  how  they  extract  information  from  their 
environments. 

Verifying.  A  particularly  valuable  application  of  AR  systems  in  human  information 
behavior  will  come  in  the  form  of  verification  of  information.  Verifying  simply  refers  to 
checking  the  accuracy  of  the  information.  “  Augmented  reality  systems  can  provide  real-time 
fact  checking.  They  can  offer  up  counter  evidence  or  arguments  as  a  user  is  viewing  or  listening 
to  various  sources  of  information.  By  employing  AR  verification  tools,  the  claims  of  experts 
could  be  questioned.  The  advertisements  of  marketers  could  be  checked.  AR  systems  could 
provide  a  continuous  monitor  for  lies  and  false  claims.  This  could  be  one  of  the  greatest  impacts 
of  AR  systems  on  how  users  interact  with  others  and  their  environment. 

Verifying  in  a  commercial  terms  could  be  visualizing  how  an  article  of  clothing  might  fit 
or  whether  a  piece  of  furniture  would  look  in  a  room.  Already  today,  these  types  of  AR 
applications  exist.  Customers  can  use  AR  fitting  kiosks  to  display  an  article  of  clothing  virtually 
120  Ellis,  1987. 
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and  then  compare  how  it  would  look  on  them.  Likewise,  using  a  smartphone  or  tablet 
prospective  furniture  shoppers  can  view  on  their  device  how  a  piece  of  furniture  would  fit  in  a 
given  space  by  pointing  the  optical  camera  of  their  device  toward  the  region  of  interest.  In  this 
manner,  AR  systems  can  better  help  their  users  visualize  and  verify  information. 

In  summary,  the  use  of  AR  systems  will  revolutionize  how  humans  learn  and  seek  out 
new  information.  The  growing  symbiosis  between  humankind  and  technology  will  only 
accelerate  when  the  barriers  to  interaction  fade  away.  With  AR  devices  that  capture  the  same 
sensory  signals  that  their  users  experience,  various  AR  applications  will  begin  to  tailor  the 
delivery  of  useful,  enhancing  information  to  their  users.  Furthermore,  the  more  AR  users  interact 
with  AR  systems,  the  more  these  systems  will  understand  their  users  unique  needs  and  desires. 
AR  systems  will  thus  provide  the  next  iterative  step  in  closing  the  distance  between  the  real- 
world  and  digital  domains.  They  will  decrease  the  time  it  takes  to  access  digital  information  and 
they  will  increase  the  amount  of  data  available  to  both  AR  users  and  AR  digital  services. 

Augmented  Reality  and  Decision-making 

The  use  of  AR  systems  will  not  only  impact  a  user’s  sensory/perceptual  experiences  and 
change  how  they  seek  out  and  use  information,  it  will  also  affect  how  AR  users  make  decisions. 
Cognitive  researchers  have  concluded  that  two  discrete  systems  exist  in  the  human  mind  that 
shape  decision-making.  One  is  based  on  intuition  the  other  on  deliberate  analysis.  These 
distinct  systems  have  been  labeled  System  1  and  System  2.  System  1  is  fast,  instinctual,  heuristic 
based,  and  emotionally  driven.  It  is  the  default  means  of  making  decisions.  It  requires  the  least 
effort  and  is  very  efficient.  System  2,  in  comparison,  is  slow,  deliberate,  and  rational.  It  is  needed 

121  Daniel  Kahneman,  Thinking,  Fast  and  Slow.  (Farrar,  Straus  and  Giroux,  2013).  See  also  K.  E.  Stanovhich  and  R. 
F.  West,  "Individual  Differences  in  Reasoning:  Implications  for  the  Rationality  Debate,"  ( Behavioral  &  Brain 
Sciences  645-665,  2000). 
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in  unfamiliar  situations  and  is  used  to  solve  complex  problems.  “  Augmented  reality  systems 
will  affect  both  reflexive  and  deliberate  forms  of  decision-making. 

As  previously  discussed,  AR  systems  will  first  shape  both  systems  of  decision-making 
through  its  effects  on  sensory/perceptual  experiences  and  information  behavior.  The  greatest 
potential  impact  of  AR  on  decision-making,  however,  could  come  in  its  users  increasing 
dependence  on  AR  systems  to  guide  their  decision-making.  These  effects  can  be  understood  by 
considering  how  dependent  most  smartphone  users  have  become  on  their  device’s  navigational 
applications.  Before  GPS  devices  and  smartphones,  people  developed  navigational  skills,  and 
made  decisions  while  traveling  based  on  a  set  of  environmental  signals.  They  used  System  1  to 
respond  to  rapidly  changing  environmental  cues  and  they  relied  upon  System  2  to  make 
deliberations  when  confronted  with  ambiguous  navigational  information.  Today,  navigation 
applications  on  smartphones  draw  drivers’  attention  away  from  their  environment  and  traditional 
navigational  indicators.  Through  smartphones,  directions  are  presented  on  2D  digital  maps  that 
update  a  user’s  location  and  generate  directions  as  they  travel.  Drivers  no  longer  have  to  pay 
close  attention  to  environmental  markers.  Nor  do  they  have  to  reason  about  their  route.  They 
follow  a  set  of  cues  with  little  cognitive  effort.  It  is  highly  likely  that  other  tasks  that  require 
intense  focus  and  concentration  for  decisions  will  be  outsourced  to  automated  decision-making 
aids  found  in  future  AR  systems. 

Augmented  reality  systems  will  change  System  1  decision-making  by  modifying  the 
heuristics  System  1  uses  to  base  its  decisions.  Heuristics  allow  users  to  quickly  categorize  a 

nr 

situation  and  make  judgments  based  on  their  previous  experiences.  They  allow  for  quick, 


122  William  Duggan,  Coup  D'Oeil:  Strategic  Intuition  in  Army  Planning,  ( Army  Strategic  Studies  Institute,  2005). 

123  Gary  A.  Klein,  "Naturalistic  Decision-making,"  (Human  Factors,  Vol.  50,  Issue  3,  2008),  pp.  456-460. 
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effective  decisions  but  they  can  also  lead  to  cognitive  biases.  '  Just  as  the  use  of  smartphone 
navigational  applications  caused  travelers  to  attend  to  alerts  on  their  mobile  devices  versus  road 
signs  in  their  environment  so  too  will  AR  systems  shift  users’  attention  away  from  other  cues  in 
their  environment  upon  which  they  once  relied  to  make  rapid  decisions. 

Augmented  reality  systems  will  have  an  even  greater  impact  on  System  2  decision¬ 
making.  To  the  extent  that  AR  users  desire  to  make  more  rational,  logical  decisions,  AR  systems 
can  search  out  and  present  them  with  information  and  perspectives  to  help  them  make  a 
determination.  Augmented  reality  systems,  connected  to  the  internal  and  other  external  data 
sources,  can  fill  gaps  in  its  users’  knowledge  and  understanding.  The  ever-growing  body  of 
collective  knowledge  found  over  the  internet,  ever  available  to  AR  users,  will  provide  a  reservoir 
of  knowledge  from  which  to  draw.  Furthermore,  the  ability  of  AR  systems  to  adapt  to  the 
idiosyncratic  information  behavior  and  preferred  leering  style  will  permit  them  to  present 
information  in  a  manner  most  beneficial  to  sound  reasoning  and  intelligence  decision-making. 
There  may  come  a  time  where  AR  users  fear  making  a  significant  decision  without  their  AR 
device  as  much  as  they  fear  traveling  to  a  new  city  today  without  their  smartphone. 


124  Daniel  Kahneman  and  Amos  Tversky,  "Judgment  Under  Uncertainty:  Heuristics  and  Biases."  ( Science  1124- 
1131,  1974).  See  also  Raymond  A.  Nickerson,  "Confirmation  Bias:  A  Ubiquitous  Phenomenon  in  Many  Guises." 
Review  of  General  Psychology,  1998)  pp.  175-220.  As  well  as  D.  Kahneman,  A.B.  Krueger,  D  Schkade,  N  Schwarz, 
and  AA  Stone,  "Would  You  be  Happier  if  You  Were  Richer?  A  Focusing  Illusion,"  ( Science  2006),  pp.  1908-1910. 
And  also  A.  Tversky  and  D.  Kahneman,  "The  Framing  of  Decisions  and  the  Psychology  of  Choice,"  ( Science , 
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Augmented  Reality  Applied  at  the  Tactical  Level 


The  following  section  will  consider  specific  applications  of  AR  technologies  at  the 
tactical  level  of  operations  in  the  military.  Building  on  the  previous  discussions  of  the  effects  of 
AR  on  users’  attention  and  its  impacts  on  learning  and  decision-making,  this  section  will  explore 
some  of  the  existing  tactical  use-cases  of  AR  systems  as  well  as  future  potential  use-cases.  The 
first  tactical  use-case  most  relevant  to  the  military  concerns  tactical  combat  operations. 

During  a  tactical  engagement,  AR  systems  can  increase  the  speed  at  which  forces  can 
orient  toward  an  adversary  and  apply  a  weapon  system.  They  can  improve  one’s  chances  during 
combats  competitive  decision-making  cycle.  John  Boyd’s  theoretical  model,  which  coined  the 
term  O-O-D-A  Loop,  describes  this  process.  According  to  Boyd,  in  tactical  engagements, 

i  oc 

operators  first  observe  what  is  going  on  around  them.  They  then  orient  this  information  and  set 
of  circumstances  to  situations  with  which  they  are  most  familiar.  To  do  this  they  rely  on  top- 
down  cognitive  processes  such  as  long-term  memories.126  After  processing  this  information, 
operators  then  make  a  decision  and  either  act  or  react.  After  this  action  the  cycle  begins  again. 
AR  systems  can  improve  each  of  these  stages. 

Observation  and  Orientation 

Augmented  reality  systems  can  aid  in  tactical  observation.  They  can  achieve  this  by 
receiving  feeds  from  sensors  on  the  battlefield  and  by  providing  their  own  sensor  inputs  back 
into  these  systems.  In  today’s  military  philosophy  of  network  centric  warfare,  all  combat  systems 
share  information  to  and  from  the  battlefield.  Armed  with  this  information,  combat  forces  can 

123  John  Boyd,  "Organic  Design  for  Command  and  Control,"  (Airpower  Australia.  May.  Accessed  May  15,  2015. 
http://www.ausairpower.net/JRB/c&c.pdf,  1987). 
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decide  whether  to  move  toward  an  enemy  or  remain  outside  of  the  effective  ranges  of  their 
weapon  systems.  Fixating  a  target  is  the  first  essential  task  in  achieving  information  superiority 
over  an  adversary  and  AR  systems  offer  the  next  expansion  of  information  technologies  capable 
of  benefiting  this  requirement. 

In  tactical  environments  combat  forces  not  only  seek  to  observe  enemy  targets  they  also 
wish  to  remain  aware  of  friendly  forces  and  civilians  within  a  given  operating  area.  The  “fog  of 
war”  makes  it  difficult  to  keep  track  of  all  of  these  forces  concurrently.  By  displaying  the  known 
locations  of  friendly  forces  and  civilian  areas  of  interest  AR  systems  aid  their  users  in  maintain 
their  situational  awareness.  As  AR  users  move  throughout  an  environment,  whether  by  land,  air, 
or  sea,  AR  systems  can  continuously  display  and  update  this  targeting  information  to  direct 
engagements  and  avoid  collateral  damage  and  fratricide. 

Tactical  Situational  Awareness 

As  combat  forces  move  through  a  tactical  environment  they  continuously  seek  to  gain 
and  maintain  SA.  Augmented  reality  systems  can  help  combat  forces  to  gain  and  maintain 
Tactical  SA  more  effectively  by  continuously  registering  information  from  their  surroundings 
and  combining  it  with  external  virtual  information  provided  by  various  battlefield  management 
and  intelligence  systems.  In  this  manner,  AR  can  increase  the  information  superiority  for  combat 
forces. 

Battlefield  management  systems  integrate  various  sources  of  combat  data  based  on  the 
operational  functions  concerning  command  and  control  (C2),  movement  and  maneuver  (M2), 
fires,  intelligence,  sustainment,  and  protection.  Most  battlefield  management  systems  provided 
automated  reporting  and  standardized  geographical  overlays  to  help  present  a  common  operating 
picture  (COP)  to  tactical  and  operational  level  units.  They  are  meant  to  aid  targeting  and 
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decision-making  cycles.  AR  systems  can  leverage  their  data  repositories  while  simultaneously 
providing  real-time  inputs  during  combat  operations.  Access  to  this  information  should  make 
combat  forces  more  effective  and  more  efficient  at  closing  with  and  defeating  an  adversary. 

Navigation  and  Maneuver 

As  operators  move  throughout  an  environment  they  must  continually  remain  oriented  to 
their  surroundings  while  attempting  to  find  the  most  effective  path  to  their  tactical  objective. 
Augmented  reality  applications  have  demonstrated  they  can  aid  navigation.  ~  They  can  provide 
enhanced  information  about  a  user’s  location  and  provide  real-time  directions  by  overlaying 
directional  cues  on  one’s  FOV.  As  previously  described,  AR  navigational  applications  aid  the 
underlying  spatial  cognitive  cues  required  for  navigation,  such  as  cues  for  orientation,  location, 
and  direction. 

A  number  of  AR  navigational  applications  for  navigation  already  exist.  GPS  enabled  map 
services  provided  by  Google  and  Apple  on  most  smartphones  represents  a  profound  AR  aid  for 
wayfinding.  In  tactical  settings,  the  ability  to  use  these  core  systems  with  specific  intelligence 
produced  overlays  that  represent  the  previous  passage  of  friendly  units  or  historical  threats  would 
allow  small-units  to  plan  their  routes  to  minimize  the  predictability  of  their  movements  and  avoid 
threats.  Beyond  tactical  mapping  software  that  provide  similar  capabilities,  AR  can  present  this 
information  in  real-time,  projected  over  a  given  area,  so  that  its  users  can  maintain  better  SA  of 
this  data  and  better  envision  their  environments. 


128  Mehdi  Mekni  and  Andre  Lemieux,  “Augmented  Reality:  Applications,  Challenges,  and  Future  Trends,”  (Applied 
Computer  Science,  2011),  pp.  205-214. 
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Targeting  and  Fires 


Augmented  reality  systems  can  also  help  combat  forces  in  the  precise  targeting  of  both 
direct  and  indirect  fires.  The  employment  of  combat  weapons  systems  requires  accurate  aim  and 
controlled  fire.  Once  an  enemy  target  is  observed  and  fixed  combat  forces  then  seek  to  apply  the 
right  military  means  to  achieve  their  tactical  objective.  AR  systems  can  recommend  weapons  of 
choice  and  in  the  future  they  may  be  integrated  with  the  actual  firing  mechanisms  of  combat 
systems.  For  example,  just  as  the  F-35  helmet  links  into  fire  control  systems  in  the  F-35,  some 
day  HMDs  worn  by  ground  forces  may  communicate  with  vehicle  mounted  weapon  systems. 

Augmented  reality  systems  can  help  operators  achieve  their  tactical  objectives  by  helping 
guide  lethal  and  non-lethal  effects  onto  a  target.  In  combat  operations  most  assume  that  operators 
would  simply  seek  to  destroy  the  target,  but  there  are  a  number  of  scenarios  in  which  an  operator 
would  rather  neutralize,  disrupt,  or  simply  deny  an  adversary  the  use  of  a  given  target. 

Depending  on  the  nature  of  the  combat  system,  the  AR  display  device  may  be  able  to 
demonstrate  what  the  potential  effect  of  that  system  may  be  on  the  surrounding  area  before  a 
given  munition  is  selected.  This  in  turn  may  help  combat  forces  save  rounds,  and  reduce 
unintended  effects.  In  addition,  using  AR  guiding  mechanisms  for  “smart  weapons,”  combat 
forces  may  achieve  levels  of  accuracy  in  fires  previously  reserved  for  guided  missile  systems. 
Smart  weapons  can  tag  targets  and  share  this  information  with  one  another.  The  scopes  on  these 
systems  also  include  sensors  that  can  collect  imagery  and  ballistic  information.  The  U.S.  Army  is 
rumored  to  have  recently  begun  testing  current  smart  rifles  with  the  goal  of  fielding  them  in  the 
near  future.  The  use  of  AR  systems  to  guide  these  direct  fire  systems  at  the  small-unit  level, 


P9  Brendan  McGarry,  “U.S.  Military  Begins  Testing  ‘Smart’  Rifles,”  (DefenseTech,  January  15,  2014, 
http://defensetech.org/2014/01/15/u-s-military-begins-testing-smart-rifles/,  accessed  October  17,  2015). 


54 


combined  with  its  ability  to  guide  indirect  fires  platforms  could  provide  an  unprecedented  level 
of  target  discrimination  and  accuracy. 

Enhanced  Understanding 

Beyond  helping  combat  forces  gain  tactical  SA  and  better  target  an  adversary,  the  use  of 
AR  in  combat  environments  will  help  combat  forces  better  understand  their  environments  and 
the  people  and  objects  with  which  they  interact.  AR  will  achieve  this  by  providing  explanatory 
information  about  locations,  objects,  and  individuals  to  which  its  users  attend.  A  variety  of  AR 
developers  have  begun  to  introduce  this  type  of  enhancing  capability.  For  example,  one  AR 
device  already  in  use  provides  enhancing  information  based  on  what  is  being  displayed  on  a 
user’s  TV  screen.  SeeSpaces’s  InAIR,  AR  software  layers  additional  content  from  the  internet 
and  social  media  in  either  2D  or  3D  formats  based  on  the  content  being  shown.  Through  this 
process  viewers  can  learn  extra  details  about  products  and  people  that  appear  in  a  television 
production.  Similarly,  Layar’s  AR  software  provides  enhancing  digital  content  to  generate 
interactive  print.  Layar  users  witness  enhancing  virtual  content  as  they  interact  and  engage  with 
traditional  print  and  other  forms  of  commercial  packaging. 

Beyond  providing  enhancing  information  to  digital  and  print  media,  AR  software 
developers  have  also  engineered  a  number  of  retail  AR  applications  that  provide  explanatory 
information  for  users  as  they  move  through  environments  and  interact  with  objects.  Retailers 
hope  to  not  only  attract  patrons  to  their  stores  through  the  use  of  augmented  reality,  but  they  also 
want  to  leverage  this  technology  to  increase  their  sales.  '  To  do  this,  they  have  begun  to  adapt 
AR  to  improve  the  shopping  experience  of  their  customers  both  in  stores  and  at  home.  They  have 

130  Philipp  Spreer  &  Katrin  Kallweit,  “Augmented  Reality  in  Retail:  Assessing  the  Acceptance  and  Potential  for 
Multimedia  Product  Presentation  at  PoS,”  (SOP  Transactions  on  Marketing  Research,  Vol  1.,  No.  1,  May  2014),  pp. 
23-31. 
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introduced  virtual  fitting  rooms,  AR  advertising,  AR  product  previews  and  enhanced  product 
information  at  the  point  of  sale.  AR  applications  exist  that  will  provide  wiki-type  content  near 
an  object  or  location  in  order  to  provide  a  potential  customer  more  information  about  a  given 

i  ri.r) 

store  or  its  product. 

Beyond  these  early  AR  applications,  mostly  developed  for  specific  commercial  clients 
and  tailored  to  provide  vendor  generated  content,  next  generation  AR  applications  will  employ 
advanced  recognition  software  to  identify  and  then  search  for  user  specified  enhancing 
information.  Object  recognition  is  one  of  the  technical  challenges  currently  limiting  AR. 
However,  the  ability  of  visual  recognition  processing  applications  to  identify  objects  in  an 
environment  from  multiple  angles  and  under  degraded  environment  conditions  is  increasing.134 

As  additional  capabilities  to  provide  enhancing  information  appear  in  commercial  AR 
systems  AR  technologies  will  be  made  available  for  military  use.  A  tactical  AR  user  today  might 
be  able  to  look  along  a  city-street  and  see  virtual  information  detailing  the  names  of  city  streets, 
building  names,  and  key  landmarks.  Soon,  military  data  sources  may  be  generated  to  provide 
information  about  sources  for  cover  and  concealment,  or  information  detailing  historical 
intelligence  reports  about  a  given  area.  One  useful  feature  already  present  in  some  AR  systems  is 
the  ability  to  have  an  X-Ray  type  view  where  an  AR  user  can  see  buildings  and  other 
environmental  information  that  is  occluded  by  other  objects  projected  virtually.  This 
information  helps  combat  forces,  as  well  as  first  responders,  to  navigate  through  man-made 

131  Philipp  Spreer  &  Katrin  Kallweit,  “Augmented  Reality  in  Retail:  Assessing  the  Acceptance  and  Potential  for 
Multimedia  Product  Presentation  at  PoS,”  (2014). 

132  Ibid. 

133  Kipper  and  Rampolla,  Augmented  Reality:  An  Emerging  Technologies  Guide  to  AR.  (2013).  See  also  Ihsan 
Rabbi,  Sehat  Ullah,  and  Siffat  Ullah  Khan,  “Augmented  Reality  Tracking  Techniques:  A  Systematic  Literature 
Review  Protocol,”  (IOSR  Journal  of  Computer  Engineering  (IOSRJCE),  Vol.  2,  (2),  2012),  pp.  22-29. 

134  David  Randall,  “Army  Heads  Up  Situational  Awareness  Technologies,”  (MILCOM  Presentation,  October  26, 
2015). 

135  Livingston  et  al.,  “Military  Applications  of  Augmented  Reality,”  (2013). 
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structures.  As  more  commercial  AR  applications  such  as  these  come  to  market  they  will  make 
their  way  into  military  settings. 

Biometrics  and  Emotional  Intelligence 

The  use  of  AR  technologies  to  expand  the  range  of  human  sensation  and  perception  is 
one  of  the  more  exciting  and  potentially  disturbing  uses  of  AR.  Currently,  sensors  on  some  AR 
systems,  like  the  Microsoft  HoloLens  can  detect  and  identify  facial  expressions  and  even  detect 
changes  in  skin  temperature  and  heart  rate.136  The  depth  camera  employed  by  HoloLens  can 
interpret  this  data  and  use  it  to  predict  the  mood  of  anyone  the  AR  user  looks  at.  For  those  that 
suffer  from  a  distinct  ability  to  detect  the  emotional  states  of  others  this  type  of  technology  could 
become  essential.  It  also  has  the  potential  to  serve  as  a  real-time  lie-detector  system  capable  of 
interpreting  the  body  language,  facial  expressions,  and  autonomic  nervous  system  responses  to 
discover  a  person’s  physiological  “tells”. 

Augmented  reality  systems  can  also  register  other  biometric  information  and  then  use  this 
information  to  identify  individuals  and  call  forth  any  available  enhancing  information  about 
them.  Before  long,  AR  cameras  and  sensors  will  be  able  to  go  beyond  simple  facial  recognition 
and  soon  be  able  to  identify  more  distinct  biological  markers  such  as  iris  distinctions  and  finger 
prints  from  a  distance.  Individuals  will  either  be  identified  in  real-time  or  selectively  registered. 

If  registered,  their  unique  biometric  markers  will  be  automatically  uploaded  and  linked  to  meta¬ 
information  such  the  location  and  nature  of  their  enrollment.  Once  the  use  of  AR  technologies 
become  adopted  widely  it  will  be  very  difficult  for  individuals  to  remain  anonymous. 


136  Mark  Hachman,  “Microsoft  Patents  Mood-Sensing  Tech  for  the  HoloLens  Augmented-Reality  Headset,” 
(PCWorld,  April  28,  2015,  http://www.pcworld.com/article/2915861/microsoft-patents-mood-sensing-tech-for-the- 
hololens-augmented-reality-headset.html,  accessed  July  20,  2015). 
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Maintenance  and  Technical  Help 


Virtual  telepresence  and  remote  collaboration  are  other  practical  uses  of  augmented 
reality.  Remote  collaboration  involves  two  or  more  people  working  on  the  same  task  using  a 

1  TO 

computer  to  mediate  communication  and  interaction.  Users  of  this  type  of  AR  system  can 
share  a  view  of  their  work  space  with  one  another.  The  value  of  this  capability  increases 
particularly  when  specialists  are  few  in  number  and  high  in  demand.  Through  telepresence  and 
remote  collaboration,  AR  can  provide  the  means  to  increase  access  to  expertise. 

The  field  witnessing  the  strongest  motivation  to  adopt  AR  is  medicine.  The  limited 
number  of  medical  specialists  combined  with  the  desire  to  simultaneously  visualize  medical  data 
and  a  patient  in  the  same  space  are  primary  drivers  of  advances  in  AR  medical  systems.  AR 
affords  doctors  the  ability  to  watch  procedures  remotely  and  then  project  their  responses  back 
into  the  FOV  of  an  onsite  technician.  Other  specialized  technicians  can  leverage  AR’s  ability  to 
provide  remote  assistance.  For  example,  Microsoft’s  recently  announced  Holo  device 
demonstrated  a  vignette  in  which  a  plumber  walked  a  customer  through  replacing  a  drain  pipe  on 
her  bathroom  sink.  The  customer  not  only  saw  a  VTC  image  of  the  plumber  projected  in  her 
FOV,  but  the  plumber  also  could  render  a  virtual  3D  diagram  in  the  customer’s  visual  perception 
of  her  bathroom  which  directed  her  how  to  properly  install  the  given  pipe. 

The  use  of  AR  systems  for  maintenance  and  engineering  applications  has  also  received  a  lot 
of  attention.  This  is  due  in  large  part  to  the  use  of  systems  produced  by  large  manufacturing 


138  Matthew  Tait,  Tony  Tsai,  Nobuchika  Sakata,  Mark  Binnghurst,  Elina  Vartiainen,  “A  Projected  Augmented 
Reality  System  for  Remote  Collaboration,”  (Presentation  at  IEEE  International  Symposium  on  Mixed  and 
Augmented  Reality,  Adelaide,  Australia,  October  1-4,  2013). 

139  Mekni  and  Lemieux,  “Augmented  Reality:  Applications,  Challenges,  and  Future  Trends,”  (2011). 
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companies  like  Boeing,  Lockheed  Martin,  BAE,  BMW,  and  Caterpillar  amongst  many  others.140 
Mechanics,  engineers,  and  assembly  line  operators  employ  AR  devices  to  guide  their  interaction 
with  various  parts  and  components.141  Rather  than  having  to  look  away  from  their  work  to 
reference  a  manual  they  see  AR  overlays  in  their  FOV.  Tests  conducted  to  determine  the  benefits 
of  using  AR  devices  reveal  that  engineers  worked  up  to  30  percent  faster  with  accuracy  at  nearly 
96  percent.  Military  mechanics  and  engineers  in  each  of  the  services  could  benefit  from  such 
tools.  Naval  engineers  working  throughout  a  ship  could  be  able  to  not  only  see  virtual  descriptors 
of  existing  systems  they  could  also  gain  the  ability  to  see  through  the  walls  and  view  future 
mechanical  installations.  Likewise,  vehicle  mechanics  could  diagnose  engine  problems  more 
rapidly  and  complete  repairs  more  efficiently  if  they  had  the  help  from  a  virtual  personal  AR 
assistant. 

In  tactical  settings,  an  operator’s  ability  to  interact  with  a  remote  technician  in  real-time 
could  dramatically  increase  their  effectiveness.  AR  could  provide  an  operator  to  see  the 
directions  of  a  subject  matter  expert  virtual  presented  in  his  FOV  thereby  increase  his 
understanding  dramatically  and  improve  the  likelihood  he  could  complete  a  given  task  for  which 
he  had  little  or  no  previous  training.  If  an  operator  had  to  treat  a  casualty  he  could  call  upon  a 
remote  doctor  to  guide  his  application  of  first  aid.  Field  mechanics  that  used  to  hunt  through 
technical  manuals  could  have  subject  matter  experts  provide  directions  through  3D  virtual 
gestures  and  instructions.  One  can  imagine  a  scenario  out  of  a  typical  Hollywood  action  film  in 


140  Sarah  A.  Webster,  “The  Fourth  Industrial  Revolution  -  It’s  Flere,”  (Huff  Post,  September  25,  2015, 
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2015). 
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which  an  operator  could  be  shown  virtually  which  wire  to  cut  on  an  explosive  device  while  a 


team  of  EOD  experts  thousands  of  miles  away  guide  him  through  his  AR  device.  Through  the 
use  of  AR  technologies,  the  U.S.  military  in  general  could  begin  to  centralize  its  various 
communities  of  expertise  and  then  distribute  this  knowledge  to  the  edge  of  the  battlefield  through 
various  AR  applications  developed  to  create  a  virtual  tele-experience. 

Piloting  and  Robotics 

As  noted  during  the  introduction  to  AR  technologies  HUDs  have  been  used  by  pilots  for 
decades.  All  military  vehicle  operators  face  the  challenges  of  maintain  oriented  to  their 
environment  as  they  attempt  to  navigate  and  maneuver.  Well-designed  AR  applications  can 
provide  this  navigational  information,  while  also  providing  information  useful  to  help  vehicle 
operators  and  their  crew  maintain  their  SA.  For  example,  pilots  of  the  USMC  AH-1Z  and  Joint 
Strike  Fighter  and  soon  Mini-Cooper  drivers,  can  see  through  the  skin  of  the  vehicles  with  AR 
devices.  By  inputting  live  video  captured  from  external  vehicle  cameras,  these  systems  provide 
pilots  increased  awareness  of  objects  in  their  environment. 

Augmented  reality  can  also  be  used  to  interact  with  and  control  robotic  and  unmanned 
systems.  Medical  and  manufacturing  robots  have  long  been  used  with  AR  devices  and 
applications.141  Robots  have  typically  been  controlled  tele-operationally  with  a  user  relying  on 
visual  cues  from  the  robot’s  perspective.  Users  have  a  difficult  time,  however,  maintaining  SA 
using  this  method.  Researchers,  therefore,  have  begun  to  explore  the  ability  of  multimodal  AR 
interfaces  to  increase  performance  in  a  user’s  ability  to  control  robotic  devices.  Their  results 


143  Paul  Milgrim,  Shumin  Zhai,  David  Drascic,  Julius  J.  Grodski,  “Applications  for  Augmented  Reality  for  Human- 
Robot  Communication,”  (Proceedings  of  the  1993  IEEE/RSJ  International  Conference  on  Intelligent  Robots  and 
Systems,  Yokohama,  Japan,  July  26-20,  1993). 
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have  found  AR  spatial  cues  benefit  users’  accuracy  and  speed  in  controlling  robotic  devices.144 
Augmented  reality  appears  ready  to  provide  the  next  generation  control  system  for  robotic 
technologies  and  unmanned  systems,  both  incredible  growth  industries. 

Training  and  Education 

The  first  use  case  for  Augmented  Reality  systems  in  the  military  that  has  gained  any 
significant  attention  has  come  from  the  training  opportunities  these  systems  provide.  A  handful 
of  research  groups,  such  as  the  Navy  Research  Lab,  have  invested  research  and  development 
efforts  to  test  the  value  of  use  AR  systems  as  training  aids.145  The  value  of  AR  systems  as 
training  aids  is  due  to  the  practical  advantages  they  provide.  Virtual  information  can  be 
combined  with  existing  live-action  training  facilities  to  reduce  changes  to  infrastructure 
requirements  and  demands  for  human  role  players.  Virtual  enemy  targets  can  be  displayed  in 
realistic  3D  depictions.  These  avatars  can  then  be  easily  reprogrammed  to  interact  in  novel  ways 
so  that  trainees  can  run  through  live-action  training  exercises  numerous  times. 

The  Marine  Corps  recently  announced  it  has  developed  an  AR  training  system  called 
Augmented  Immersive  Team  Trainer  (AITT).146  This  system  allows  marines  to  train  on  directing 
airstrikes  and  calls-for  indirect  fire.147  Typically  this  type  of  training  requires  intense 
coordination  for  military  assets  and  costs  a  lot  due  to  the  expense  of  these  types  of  munitions.  By 
providing  realistic  training  scenarios,  AR  systems  such  as  AITT  can  reduce  these  costs  and 
increase  opportunities  for  combat  forces  to  increase  their  proficiency  in  complex  tactical  tasks. 
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Challenges  and  Recommendations  for  Adopting  AR  Technologies  at  the  Tactical 

Level 


As  beneficial  as  AR  systems  may  before  for  actions  taken  at  the  tactical  level,  there  are  a 
number  of  challenges  that  will  have  to  be  overcome  before  any  military  accepts  their  whole- sale 
use.  Issues  concerning  safe-use  practices  are  a  primary  concern.  In  addition,  certain  technical 
limitations  in  AR  systems  currently  exist  that  may  create  some  hesitancy  in  procurement.  Lastly, 
a  handful  of  social  issues,  privacy  concerns,  and  unique  security  concerns  of  the  military  will 
have  to  be  addressed  before  AR  systems  are  allowed  within  the  ranks  of  any  branch  of  service. 

Distraction 

Balancing  the  level  of  distraction  of  AR  technologies  will  be  a  fundamental  challenge 
and  consideration  in  the  evolution  of  AR  system  designs  and  control  features.  Augmented 
reality  systems  can  automatically  capture  the  attention  of  their  users  by  presenting  particularly 
salient  signals  and  cues  within  their  FOV.  As  was  noted  in  the  previous  discussion  concerning 
the  impacts  of  AR  systems  on  sensory/perceptual  experiences,  these  signals  can  generate 
distractions  a  user  may  not  be  able  to  avoid. 

To  overcome  the  potential  safety  issues  that  could  result  from  AR  systems  distractors  the 
first  step  should  be  the  development  of  built-in  safety  measures.  The  systems  should  have 
automatic  kill- switches  that  disable  the  presentation  of  any  distracting  information  both  manually 
and  automatically  in  times  of  great  distress.  In  addition,  AR  systems  designers  should  refer  to 
references,  such  as  the  Oxford  Handbook  for  Cognitive  Engineering,  that  provide  recommended 
best  practices  in  system  design  that  take  into  account  how  the  human  mind  functions.  Finally, 

148  Kipper  and  Rampolla,  Augmented  Reality:  An  Emerging  Technologies  Guide  to  AR.  (2013) 
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future  AR  systems  should  themselves  offer  alerts  and  warnings  to  their  users  when  a  danger 
appears  and  take  actions  to  focus  a  user  on  responding  to  such  situations. 

Sensor  Range  and  Accuracy 

A  technical  limitation  of  current  AR  systems  is  the  range  and  accuracy  of  their  intrinsic 
sensors.  To  the  extent  that  these  sensors  are  used  to  collect  environment  data  their  sensitivity  will 
impact  the  quality  of  later  AR  processing,  and  rendered  virtual  information.  Bad  data  in  leads  to 
bad  data  out.  In  tactical  military  settings  this  may  have  grave  effects. 

Fortunately,  sensors,  like  micro-processors,  appear  to  be  following  the  same  development 
path  predicted  by  Moore’s  Law.  Nearly  every  18  to  24  months,  they  gain  increased  sensitivity 
and  accuracy,  while  decreasing  in  size,  cost,  and  energy  consumption.  For  military  AR  systems 
developers  and  engineers,  this  means  that  the  current  semi-bulky,  non-ruggedized,  large-energy 
consumers  may  transform  into  much  smaller,  much  lighter,  more  rugged,  sustainable  systems. 

Connectivity  at  the  Edge  of  the  Battlefield 

One  limiting  factor  for  AR  system  performance  in  combat  operations  is  their  access  to 
external  data  sources.  Connectivity  to  the  edge  of  the  battlefield  remains  a  central  problem  for 
Signal  planners  across  all  branches  of  service.  Over  the  past  two  decades,  as  the  military  has 
operated  in  regions  with  little  or  no  internet  connectivity,  they  have  engineered  systems  that  can 
now  allow  them  to  provide  internet  connectivity  to  deployed  forces.  The  terminus  for  these  links 
typically  resided  on  large  forward  operating  bases  with  static  satellite  terminal  sites.  This  started 
to  change  after  2010,  however. 

Today,  more  systems  have  been  fielded  that  provide  reach-back  connectivity  both 
through  line-of-site  and  beyond-line-of-site  data  links.  These  forward  deployed  tactical  networks 
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can  be  leveraged  by  tactical  AR  systems.  Through  secure  wireless  links,  AR  systems  can  push 
and  pull  data  across  these  systems.  The  remaining  question  will  concern  where  does  do  the 
military  AR  data  services  reside.  If  they  stored  in  a  CONUS  location  then  the  fragility  of  the  data 
link,  its  bandwidth,  and  latency,  could  greatly  impact  deployed  AR  experiences.  If  these  data 
sources  can  be  incorporated  into  forward  deployed  battle  management  systems,  then  these 
challenges  would  be  overcome.  Future  programs  of  record  for  a  Joint  AR  system  should  take 
these  factors  into  consideration  in  their  design. 

Social  Challenges,  Privacy,  and  Data  Security 

Perhaps  the  greatest  limiting  factors  to  the  adoption  of  AR  systems  within  the  military  are 
social  challenges,  privacy  concerns,  and  unique  security  classification  of  military  information. 
The  introduction  of  a  new  technology  like  AR  can  disrupt  social  norms.  When  Google 
introduced  its  AR  devices  in  public  settings  many  of  its  users  experienced  strong  backlashes 
from  others.149  Many  see  AR  display  devices  as  barriers  to  interaction  and  forms  and  social 
distractors.  Others  are  concerned  about  their  privacy  due  to  the  cameras  on  these  devices  that 
record  and  register  AR  users’  surroundings.150 

Military  personnel  will  inevitably  share  these  concerns,  but  they  also  have  to  manage 
information  that  may  be  sensitive  or  classified.  A  key  question  for  future  military  AR  system 
designers  is  how  to  turn  on  and  off  the  collection  of  AR  sensor  data,  and  how  to  selectively 
classify  it.  For  example,  data  from  a  combat  engagement  may  need  to  be  restricted,  but  data 
about  geographical  features,  locations  of  infrastructure,  etc.  may  be  useful  for  less  sensitive  data 
services.  A  system  that  does  not  classify  all  sensor  data  in  a  bulk  manner  may  increase  the  utility 
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of  future  AR  systems.  Furthermore,  future  AR  system  designers  will  have  to  deal  with  the 
challenging  issues  of  securing  AR  data-at-rest  and  the  potential  loss  or  compromise  of  these 
devices.  Programs  exist  already  for  mobile  devices  and  laptops  that  can  effectively  erase 
sensitive  data  if  it  falls  into  the  wrong  hands.  Any  viable  military  AR  system  will  have  to  do  the 
same. 
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Augmented  Reality  Applied  at  the  Operational  Level 


The  majority  of  proposed  and  tested  use-cases  for  AR  systems  have  focused  on  their 
tactical  level  applications.  In  reality,  AR  systems  may  be  used  in  military  office  spaces  and 
conference  rooms  well  before  they  are  fielded  to  tactical  combat  forces.  The  reason  for  this,  is 
that  many  of  these  systems  are  being  designed  to  increase  individual  productivity,  creativity,  and 
collaborative  ability.  In  addition,  because  the  military  requires  much  higher  standards  for 
performance  of  information  technology  systems  designed  for  use  in  harsh  tactical  environments 
its  more  likely  the  military  will  begin  to  use  AR  systems  in  Garrison  first.  Therefore,  the 
following  section  will  explore  some  of  the  ways  AR  systems  will  alter  and  benefit  operational 
level  planning  and  decision-making. 

The  U.S.  military’s  doctrine  for  planning  recognizes  that  operational  planners  rely  on 
both  crisis  response  planning  as  well  as  more  deliberate  efforts.151  In  crises  action  planning 
(CAP),  operational  staffs  use  the  collective  experience  and  knowledge  of  their  planning  team  to 
develop  courses  of  action  rapidly.  During  deliberate  planning,  they  spend  greater  time 
researching  viable  course  of  action  and  seeking  outside  expertise.  As  previously  discussed,  AR 
systems  can  benefit  the  fundamental  decision-making  processes  behind  both  intuitive  and 
deliberate  planning.  Beyond  improving  these  forms  of  operational  decision-making,  however, 
AR  can  also  benefit  operationally  command  and  control  through  improved  operational 
situational  awareness,  better  visualization  tools,  and  improved  means  of  providing  virtual 
presence  and  communication. 


151  Karol  G.  Ross,  Gary  A.  Klein,  Peter  Thunholm,  John  F.  Schmitt,  and  Holly  C.  Baxter,  "The  Recognition -Primed 
Decision  Model,"  ( Military  Review  6-11,  2004). 
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Situational  Awareness  and  Understanding 


Augmented  reality  systems  can  improve  SA  and  understanding  through  their  ability  to 
integrate  multiple  sources  of  information  to  generate  a  common  operating  picture  and  then  share 
this  picture  through  various  battlefield  management  systems.  In  order  to  plan  and  make  decisions 
regarding  the  employment  of  tactical  forces  operational  leaders  must  understand  and  monitor  the 
Operational  Factors  of  Space,  Time,  and  Force.  They  need  to  understand  and  visualize  how  these 
factors  interact  with,  and  affect  their  forces  and  their  adversaries’  forces.  In  addition,  they  must 
remain  aware  of  the  relative  combat  potential  of  these  forces  -  typically  through  indicators 
across  the  six  Operational  Functions  (i.e.  C2,  Movement  and  Maneuver,  Fires,  Intel,  Protection, 
and  Sustainment).  Operational  leaders  use  their  understanding  of  these  elements  to  respond  in 
times  of  crisis  and  to  plan  deliberately.  By  monitoring  AR  COP  applications,  an  operational 
leader  can  stay  abreast  of  any  significant  changes  in  these  elements. 

Augmented  reality  systems  allows  users  to  see  virtual  representations  of  forces,  with 
indicators  of  combat  readiness,  projected  over  either  real  environments  or  digitally  rendered 
common-operating-pictures  (COP).  With  an  AR  display  device,  an  operational  leader,  whether 
they  are  in  their  office,  state  room,  the  DFAC,  or  out  on  the  battlefield  can  witness  the  precise 
location  and  movement  of  both  friendly  and  enemy  forces.  On  the  battlefield,  these  indicators 
may  influence  a  commander’s  understanding  of  how  a  battle  is  progressing.  For  example,  a 
commander  may  see  virtual  lines  moving  as  tactical  forces  gain  and  lose  territory.  They  may  see 
progress  on  the  field  of  battle  the  same  way  that  viewers  see  AR  first-down  lines  shift  on 
televised  football  games  today.  These  AR  overlays  provide  both  audience  the  means  to  see 
quickly  where  their  team  is  at  and  how  much  further  they  have  to  go. 
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Augmented  reality  systems  also  increase  SA  by  providing  real-time  updates  and  alerts  to 
operational  leaders.  As  battle  management  systems  collect  information  and  updates,  they  can 
rapidly  disseminate  this  information  through  AR  systems.  An  operational  leader’s  continuous 
access  to  information  about  the  availability,  readiness,  capability,  and  capacity  of  tactical  forces 
allows  him  to  determine  if  they  are  near  culmination  or  not.  This  can  increase  response  times  to 
deploy  reserve  forces  or  to  provide  medical  support.  Likewise,  the  combined  running  estimates 
of  an  enemy  situation  informs  operational  leaders  how  close  tactical  forces  are  to  achieving 
victory.  The  increased  fidelity,  timeliness,  and  relevance  of  the  information  provided  by  AR 
systems  to  operational  leaders  will  enable  them  to  better  direct  their  forces. 

Another  way  that  operational  leaders  will  benefit  will  come  through  the  nature  of  the 
enhancing,  explanatory  information  military  AR  systems  may  soon  provide.  Future  battle  update 
briefs  in  which  an  intelligence  analyst  refers  to  a  particular  enemy  target  may  trigger  an 
operational  leaders  AR  system  to  display  enhancing  information  about  the  targets  ethnic 
affiliation,  known  associates,  and  last  registered  location.  This  access  to  enhancing  information 
will  speed  up  the  sharing  of  information  and  increase  the  understanding  of  operational  leaders 
and  their  staffs. 

As  beneficial  as  it  is  for  operational  leaders  to  gain  access  to  better  information  and 
continuous  SA  updates,  the  ability  of  AR  systems  to  adapt  to  their  users’  learning  preferences 
will  prove  even  more  significant.  As  was  previously  noted,  AR  systems  will  begin  to  modify 
how  they  guide  their  users  to  information  once  they  learn  their  preferred  information  sources  and 
learning  styles.  In  addition,  they  will  likely  vary  the  aperture  of  their  search  based  on  a  user’s 
available  timeframe  to  process  it.  In  this  manner,  these  systems  will  create  a  more  adaptive  user 
experience. 
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Visualization  and  Operational  Design 


After  operational  leaders  understand  their  operational  environment  and  the  particular 
problem  they  face  they  begin  to  develop  plans.  They  start  by  formulating  a  desired  end-state  and 
then  they  visualize  various  courses  of  actions.  They  typically  apply  aspects  of  both  operational 

1  cn 

design  and  art  to  develop  a  lines  of  operation  focused  on  achieving  a  desired  objective.  The 
use  of  AR  systems  can  benefit  operational  planners  as  they  work  through  these  processes. 

First,  operational  leaders  can  use  AR  systems  to  help  them  visualize  possible  courses  of 
action.  In  the  same  manner  that  AR  systems  are  already  helping  engineers  and  manufacturers 
visualize  their  conceptions,  so  too  can  they  help  operational  leaders  see  how  the  employment  of 
tactical  forces  would  take  place  in  both  time  and  space.  '  For  example,  3D  CAD  renderings 
allow  engineers  and  architects  to  experience  diagrams  virtually  in  3D  and  to  scale.154  Operational 
leaders  could  leverage  future  battlefield  management  systems  to  generate  similar  virtual 
renderings  of  their  operational  plans.  These  renderings  could  then  be  displayed  in  AR  sand- 
tables,  such  as  that  developed  by  ARES  systems,  or  perhaps  projected  in  space  while  wearing  an 
AR  display  device.  For  example,  in  the  future,  an  operational  commander  afforded  the 
opportunity  to  conduct  an  in-person  reconnaissance  form  the  vantage  of  a  flying  platform  could 
use  an  AR  system  to  virtual  project  tactical  forces  and  visualize  how  various  tactical  encounters 
might  play  out. 


154  JCS.  JP3-0,  2011. 

153  Rana  Abboud,  "Architecture  in  an  Age  of  Augmented  Reality:  Opportunities  and  Obstacles  for  Mobile  AR  in 
Design,  Construction,  and  Post-Completion,"  (2014). 

154  Anthony  Webster,  Steven  Feiner,  Blair  MacIntyre,  William  Massie,  and  Theodore  Krueger,  n.d.  Augmented 
Reality  in  Architectural  Construction,  Inspection,  and  Renovation,  (Accessed  February  14,  2015. 
http://www.columbia.edu/cu/gsapp/BT/RESEARCH/PAPERS/ar-asce.html). 
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Describing  for  a  Shared  Vision 


After  operational  leaders  have  developed  an  operational  plan  they  then  must 
communicate  it.  Augmented  reality  systems  can  help  operational  leaders  share  their 
understanding  and  proposed  tactical  actions.  In  coordinating  up,  down,  and  between 
organizations  and  agencies,  operational  leaders  must  address  a  number  of  audiences. 155 
Augmented  reality  can  help  in  this  effort  by  accessing  and  display  data  from  battlefield 
managements  systems  to  help  present  and  organize  information  in  a  meaningful  way.  Just  as 
smartphones  and  mobile  devices  are  keeping  leaders  better  connected  to  each  other,  and 
information,  so  too  will  AR  systems  enhance  the  communication  of  shared  visions. 

Augmented  reality  can  aid  communication  and  increase  shared  understanding  by  creating 
and  maintaining  virtual  collaborative  environments  controlled  by  next  generation  battlefield 
management  systems.  The  benefit  of  AR  systems,  is  that  they  provide  timely  access  and  updates 
to  these  systems  both  in  garrison  and  tactical  environments.  There  is  no  delay  in  the  upload. 
Tactical  units  to  not  have  to  return  to  the  forward  operating  base  to  generate  and  update  a  field 
report.  This  report  can  be  generated  in  real-time,  and  shared,  through  the  use  of  AR  systems. 
Through  this  real-time  updating,  AR  systems  can  provide  a  continuously  updated  common 
virtual  operating  picture.  A  number  of  battlefield  management  systems  currently  exist  that  could 
be  modified  or  integrated  with  future  AR  technologies.  This  will  facilitate  greater  information 
sharing  and  collaboration  to  and  from  tactical  objectives  which  would  in  turn  facilitate  increased 
collaboration  and  self-synchronization.  The  realization  of  AR  systems  employed  both  at  the 


155HQDA,  ADP  5-0,  2012. 
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tactical  and  operational  levels  would  move  military  operations  closer  to  the  network  centric 
warfare  philosophy  of  power-to-the-edge.156 

Direction  and  Leading 

Augmented  reality  systems  could  also  benefit  operational  leaders  in  their  efforts  to  direct 
and  lead  forces  both  in  garrison  and  in  combat.  Regardless  of  the  information  technologies 
available,  Operational  commanders  must  describe  their  intent,  establish  achievable  objectives, 
and  issue  clear  tasks  and  guidance.157  Augmented  reality  systems  would  make  operational 
leaders  more  effective  and  efficient  in  these  tasks.  This  in  turn  would  allow  them  to  respond 
more  quickly  and  remain  more  agile.  In  combat,  controlling  the  pace  of  operations  is  integral  a 
combat  force’s  success.158  U.S.  military  forces  desire  to  operate  at  the  speed  of  the  problem}59 
AR  can  increase  an  operational  leader’s  ability  to  maintain  a  relative  speed  advantage  in 
directing  and  leading  forces  -  particularly  those  forces  that  have  come  to  rely  upon  precision 
guided,  highly  synchronized  operations  to  defeat  high-tech  adversaries. 

Increasing  the  speed  at  which  operational  leaders  can  make  decisions  does  little  without 
also  providing  the  means  to  implement  them.  During  combat,  changes  to  intent  and  objectives 
must  be  rapidly  disseminated.  Due  to  the  ability  of  AR  systems  to  continuously  receive  alerts  to 
update  a  user,  they  can  dramatically  reduce  the  time  it  takes  to  notify  key  leaders  and  personnel 
of  changes.  The  use  of  AR  systems  allows  remote  users  to  share  and  communicate  information. 
For  example,  distant  operators  could  share  their  sensor  information  so  that  an  operational  center 


156  Alberts,  The  Agility  Advantage,  (2011). 

157  HQDA,  ADP  5-0,  2012. 

158  JCS,  JF-2020,  2012. 
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could  see  first-hand  footage  of  an  engagement.160  This  is  just  another  form  of  virtual 
telepresence.  At  the  same  time,  AR  systems  provide  traditional  voice  communications  and 
instant  messaging  applications. 

Throughout  all  phases  of  planning  and  combat  operations  the  presence  of  leaders  matters. 
Face-to-face  interactions,  and  direct  understanding,  increases  cohesion  and  develops  trust  within 
organizations.161  Some  have  argued  that  the  use  information  technologies  has  caused  leaders  to 
distance  themselves.  Much  of  this,  however,  may  be  accounted  for  by  the  increased  demands 
placed  on  all  leaders  to  remain  always  connected.162  With  the  advent  of  AR  systems,  leaders 
could  still  maintain  an  eye  on  various  notifications  of  interest,  but  they  would  not  have  to  remain 
tethered  to  the  computer  in  their  office.  Augmented  reality  systems  can  help  leaders  interact  with 
their  teammates  without  worrying  about  missing  any  time-sensitive  information. 

Assessing 

As  operations  unfold,  operational  leaders  must  continuously  assess  the  situation.  They 
must  anticipate  and  adapt  operational  plans  to  changing  conditions.163  Operational  leaders  are 
expected  to  continuously  measure  the  overall  effectiveness  of  the  actions  of  their  tactical 
forces.164  Through  instant  updates  from  the  edge  of  the  battlefield,  and  increased  outputs  from 
deployed  AR  sensors,  operational  leaders  can  gain  better  assessments  of  tactical  actions. 


160  Shenai,  M.B.,  M  Dillavou,  C.  Shum,  D.  Ross,  R.S.  Tubbs,  A.  Shih,  and  B.L.  Guthrie,  "Virtual  Interactive 
Presence  and  Augmented  Reality  (VIPAR)  for  Remote  Surgical  Assistance."  (Neurosurgery,  Vol.  68,  2011),  pp. 
200-207. 

161  Stephen  M.R.  Covey,  The  Speed  of  Trust :  The  One  Thing  that  Changes  Everything.  (New  York:  Free  Press, 
2008). 

162  Nicholas  Bloom,  Luis  Garicano,  Raffaella  Sadun,  and  John  Ven  Reenan,  The  Distinct  Effects  of  Information 
Technology  and  Communication,  (CEP  Discussion  Paper  No  927,  London:  Center  for  Economic  Performance, 
London  School  of  Economics  and  Political  Science,  2009). 
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The  most  important  impact  of  AR  systems  in  improving  the  ability  of  operational  leaders 
to  assess  operations  is  from  their  production  of  timely  information.  However,  other  features  of 
AR  systems  may  better  enable  operational  leaders  to  understand  changes  to  an  operational 
environment  or  forces.  For  example,  abstract  variables  in  a  complex  adaptive  system,  such  as 
public  sentiment,  rates  of  violence,  and  perceived  threats  may  be  more  concretely  visualized  and 
analyzed  through  the  use  of  AR’s  3D  graphical  representations.  Future  AR  representations  of 
complex  battlefield  elements  may  help  operational  leaders  to  understanding  these  interacting 
dynamics.  Various  AR  systems’  use  of  concrete  images  and  representations  to  explain  complex 
topics  has  already  been  demonstrated  to  aid  learning  -  particularly  when  the  information  is 
abstract  or  difficult  to  imagine.165 

Finally,  through  their  access  to  big-data  analytical  tools,  AR  systems  will  provide  users 
powerful  real-time  analytical  aids.  Big-data  analytics  are  used  today  to  comb  through  vast 
amounts  of  data  to  discover  patterns  and  hidden  trends.  Future  AR  systems  will  not  only  have 
access  to  repositories  of  collected  data  but  will  also  likely  contribute  their  own  data  to  these 
collections.  Every  AR  user  will  be  a  sensor.  Therefore,  the  actions  of  AR  users  will  also  be 
analyzed  and  become  their  own  measures  of  effectiveness. 

Challenges  and  Recommendations  for  Adopting  AR  Technologies  at  the  Operational  Level 

The  challenges  of  adopting  AR  systems  at  the  operational  level  build  upon  the  concerns 
raised  at  the  tactical  level.  However,  some  of  the  concerns  differ  based  on  cultural  differences 
between  tactical  operators  and  operational  leaders  and  their  staffs.  At  the  tactical  level,  AR 
systems  can  transform  many  manual  tasks  and  alter  how  people  directly  interact  with  their 

165  Mehmet  Kesim,  and  Yasin  Ozarslan,  "Augmented  Reality  in  Education:  Current  Technologies  and  the  Potential 
for  Education."  (Procedia  -  Social  and  Behavioral  Sciences  297-302,  2012). 
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environments.  At  the  operational  level,  the  use  of  AR  systems  will  serve  as  an  expansion  of  the 
capabilities  found  in  today’s  IT  productivity  suites  and  battlefield  management  systems.  The 
following  concerns  may  overlap  with  some  of  the  tactical  challenges  of  adopting  AR  systems, 
but  perhaps  fit  more  of  the  central  issues  of  integrating  AR  systems  at  the  operational  level. 


Anti- Technology  Biases 


The  first  challenge  to  overcome  in  adopting  AR  technologies  within  operational  level 
staffs  and  organizations  will  be  overcoming  anti-technology  cultural  biases.  Most  operational 
leaders  and  their  staffs  are  more  senior  in  rank  and  age  to  tactical  level  operators.  As  such,  they 
will  typically  be  lest  accustomed  to,  and  comfortable  with  new  information  technologies.  At  the 
operational  level,  leaders  that  choose  to  adopt  AR  systems  will  have  to  be  aware  that  many 
organizations  suffer  from  an  anti-technology  bias  among  various  groups  of  employees.166  For 
example,  when  the  concept  of  Network  Centric  Warfare  was  introduced  in  1999,  many  high- 
profile  military  leaders  resisted  the  notion  that  the  Information  Age  would  significantly  alter  how 
the  U.S.  military  functioned.167  As  a  result,  they  were  slow  to  anticipate  the  changes  information 
technologies  would  play  on  combat  operations  and  make  the  necessary  strategic  investments  to 
fully  exploit  these  transformations. 

If  organizations  do  not  co-evolve  their  structure,  processes,  and  policies  to  fully  leverage 
advancements  in  information  technologies  they  incur  an  enormous  opportunity-cost.168  The 
introduction  of  AR  systems  will  require  the  military  to  reassess  and  restructure  its  organization, 
practices,  and  doctrine.  The  military  should  begin  to  educate  its  leaders  and  service  personnel  of 
the  capabilities  and  opportunities  of  AR  systems  now  so  they  can  fully  take  advantage  of  them 
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Karl  D.  Schubert,  CIO  Survival  Guide:  The  Roles  and  Responsibilities  of  the  Chief  Information  Officer ,  (John 
and  Wiley  Sons,  2005). 

167  Alberts,  The  Ability  Advantage,  (2011).  . 

168  Ibid. 


74 


once  they  arrive  within  their  ranks.  If  the  military  fails  to  anticipate  the  sea-change  this 
technology  will  have  on  humankind  in  general,  it  will  not  take  the  prerequisite  steps  to  plan  for 
and  build  the  infrastructure  necessary  to  make  full  use  of  AR’s  capabilities. 

Information  Overload 

A  concern  at  the  operational  level  is  that  AR  users  will  become  overwhelmed  by  the 
amount  of  information  available  to  them.169  Information  overload  can  occur  when  too  much 
information  is  available  and  is  not  well-organized.  As  some  seek  out  more  and  more 
information  they  can  begin  to  hesitate  to  act.  They  will  wait  until  the  last  piece  of  evidence  can 
be  obtained.  As  operational  leaders  become  accustomed  to  ever  present  streams  of  real-time, 
enhancing  information,  they  may  experience  paralysis-by-analysis. 

As  concerning  as  information  overload  may  be,  the  fundamental  purpose  behind  AR 
systems  is  to  focus  a  user’s  attention  to  relevant  information.  Great  IT  systems  are  designed  to 
provide  essential  information  in  an  intuitive  manner.  “  If  the  military  hopes  to  prevent 
information  overload  it  should  begin  by  specifying  requirements  for  future  AR  systems  that  can 
help  ensure  AR  systems  prevent  distraction  and  information  overload.  Additionally,  through 
training  and  education,  operational  leaders  and  their  staffs  can  leam  to  integrate  AR  systems  into 
their  decision-making  processes  in  a  manner  meant  to  mitigate  the  potential  for  information 
overload. 

An  Over  Dependence  on  Augmented  Reality 


169  Thomas  P.M.  Barnett,  "The  Seven  Deadly  Sins  of  Network-Centric  Warfare,"  (Proceedings  Magazine,  1999). 

170  Milan  Vego,  Joint  Operational  Warfare:  Theory  and  Practice.  (Govt  Printing  Office,  2009). 

171  AlanZimm,  "Human-Centric  Warfare."  ( Proceedings ,  Vol  125,  No.  5,  May,  1999). 

172  Laura  G.  Militello,  and  Gary  Klein,  "Decision-Centered  Design."  (In  The  Oxford  Handbook  of  Cognitive 
Engineering,  by  John  D.  Lee  and  Alex  Kirlik,  261-271.  New  York:  Oxford  University  Press,  2013). 
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Another  concern  is  that  decision  makers  will  become  overly  dependent  on  AR  systems 
after  they  grow  accustomed  to  their  continuous  interaction  with  these  systems.  Today,  many 
have  become  addicted  to  information  technologies.  Most  would  rather  leave  home  without  their 
wallets  than  without  their  mobile  devices.  Operational  leaders  that  begin  to  lose  their  ability  to 
make  sound  decisions  without  the  help  from  information  technologies  may  struggle  when  they 
lose  access  to  these  systems.  For  example,  if  an  adversary  disrupted,  denied,  or  destroyed  access 
to  these  information  systems  they  may  gain  a  distinct  advantage  over  future  AR  dependent 
militaries. 

As  concerning  as  an  over  dependence  on  information  technologies  may  be,  organizations 
should  not  avoid  integrating  technologies  that  may  offer  them  an  advantage  simply  because  they 
fear  someday  losing  access  to  them.  Leaders  at  all  levels  must  manage  this  risk  and  develop 
means  to  mitigate  it  by  developing  redundancy  and  alternative  means  of  commanding  and 
controlling  their  forces.  As  new  disruptive  technologies  appear  it’s  the  leader’s  place  to 
determine  how  best  to  leverage  their  capabilities  as  well  as  manage  the  new  risks  that  emerge 
after  their  adoption. 

Fears  of  Micromanagement 

Perhaps  the  greatest  fear  of  many  within  the  military  is  that  the  proliferation  of 
information  technologies  has  led  to  greater  micromanagement  by  senior  leaders.174  In  fact,  some 
believe  the  call  for  mission  command  by  senior  military  leaders  is  meant  to  address  the  growing 
concern  that  leaders,  at  all  levels,  abuse  the  capabilities  of  new  C4ISR  systems  to  increase  their 

173  Meena  H.  Duerson,  “We’re  Addicted  to  Our  Phones:  84%  Worldwide  Say  They  Couldn’t  Go  a  Single  Day 
Without  Their  Mobile  Device  in  Their  Hand,  ”  (2012). 
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to  Support  Military  Decision-making  and  Command  and  Control,"  (In  Decision-making  in  complex  environments, 
by  Malcolm  Cook,  Janet  M  Noyes  and  Yvonne  (Eds.)  Masakowski,  Aldershot:  Ashgate,  2007),  pp.  191-201. 
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access  and  influence  on  the  actions  of  their  subordinate  units.  Introducing  AR  systems  at  the 
tactical  and  operational  levels  has  the  potential  to  heighten  these  fears.  The  use  of  AR  systems 
represents  a  further  extension  of  information  technologies  into  the  daily  lives  and  activities  of 
service  personnel.  Therefore  some  will  resist  their  use. 

Micromanagement  is  a  people  problem  -  not  a  technology  problem.  Regardless  of  the 
technologies  available,  some  leaders  trust  their  subordinates  others  do  not.  Rather  than  viewing 
AR  as  a  threat  to  autonomy,  it  should  be  conceived  of  as  a  means  to  flatten  organizations  so  that 
greater  unity  of  effort  and  shared  understanding  can  take  place. 


175  P.W.  Singer,  "Tactical  Generals:  Leaders,  Technology,  and  the  Perils  of  Battlefield  Micromanagement,"  (Air  & 
Space  Power  Journal ,  2009). 
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Discussion  and  Recommendations 


Though  the  military  will  have  to  overcome  some  unique  challenges  associated  with 
integrating  AR  into  its  ranks  the  benefits  of  adopting  AR  far  out  way  the  costs.  Augmented 
reality  systems  have  demonstrated  that  they  have  reached  a  commercial  tipping  point.  Soon, 
consumers  will  begin  to  purchase  and  use  them  throughout  their  daily  lives.  From  this,  greater 
demand  signals  will  be  placed  on  planners  within  the  military  to  leverage  the  capabilities  of  AR 
systems.  The  following  set  of  recommendations  are  offered  to  help  planners  prepare  for  this 
inevitability. 

Recommendation  1  -  Anticipate  Back-end  Infrastructure  Requirements  and  Policy 
Changes 

In  order  to  take  full  advantage  of  the  potential  of  AR  systems  the  military  should  not  wait 
for  commercial  sectors  to  fully  mature  before  sharing  its  requirements  and  shaping  future  AR 
developments.  The  military  should  begin  to  capture  its  unique  needs  and  communicate  these  to 
its  defense  industry  partners  in  order  to  realize  fully,  the  maximum  potential  of  AR  systems.  It 
will  not  be  enough  to  procure  these  systems  and  distribute  them  to  units.  A  significant 
investment  in  developing  the  requisite  back-end  infrastructure  to  support  these  new  sources  of 
data  and  information  flow  must  take  place.  In  addition,  the  military  will  have  to  reconsider  many 
of  its  policies,  particularly  as  they  relate  to  data  classification,  security,  and  access.  Leaders  will 
have  to  determine  where  risk  resides  and  decide  if  combat  forces  are  being  hamstrung  by 
outdated,  overly  risk  averse  policies  that  prevent  them  from  fully  leveraging  emerging 
information  technologies  such  as  AR. 

Recommendation  2  -  Develop  a  Holistic  AR  Data  Management  Plan 
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Any  augmented  reality  system  is  only  as  good  as  the  virtual  information  it  provides.  As 
the  military  plans  for  and  investing  in  future  battlefield  management  systems  it  should  consider 
what  information  will  be  of  most  use  to  military  AR  users.  It  should  also  consider  what 
information  will  come  from  military  AR  users.  In  addition,  the  military  should  demand  that  these 
sources  of  information  are  capable  of  being  input  into  other  C4ISR  applications  regardless  of 
manufacturer.  The  growth  of  big-data  analytics,  super-computers,  and  semi-artificial  intelligent 
personal  assistants  is  only  going  to  increase.  Military  personnel  will  desire  to  leverage  these 
services  as  much,  if  not  more  than  commercial  entities.  Therefore,  the  military  must  lead  an 
effort  to  ensure  that  AR  data  remains  open-source,  interoperable,  and  standardized. 

Recommendation  3  -  Intuitive  and  Rugged 

Military  AR  systems  must  be  intuitive  to  use  and  designed  for  rugged  use.  They  should 
be  designed  to  elegantly  present  only  the  information  that  enhances  a  user’s  perceptual 
experience.  They  should  deliver  simple  means  for  decision-making  and  increase  a  user’s  ability 
to  communicate  and  share  information.  They  should  not  be  a  computer  monitor  mounted  to  a 
helmet.  Augmented  reality  systems  cannot  become  distractions  or  difficult  to  operate  or  they  will 
be  discarded  quickly.  The  military,  and  its  defense  industry  partners,  should  take  a  note  from 
corporations  like  Apple  devote  as  much  time  in  engineering  elegant  interfaces  as  they  do  in 
writing  software  code.  At  the  same  time,  the  same  set  of  AR  glasses  that  a  service  member  uses 
in  an  office  setting  will  not  hold  up  in  the  field.  Any  additional  piece  of  kit  worn  in  combat  must 
be  lightweight,  rugged,  and  unobtrusive.  These  systems  should  replace  a  required  piece  of 
equipment  if  at  all  possible.  For  example,  tactical  AR  systems  should  replace  required  eye- 
protection  as  well  as  other  optics  such  as  night-vision  or  thermal  lenses.  Combat  forces  should 
not  have  to  swap  between  multiple  pairs  of  glasses  if  it  can  be  avoided. 
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Recommendation  4  -  New  Organization  Structures  and  Military  Occupations 

Finally,  the  military  should  begin  to  plan  for  the  changes  in  organizational  structure, 
processes,  and  doctrine  that  will  occur  after  the  proliferation  of  AR  systems  within  the  ranks  of 
the  military.  The  military  must  consider  how  to  transform  its  force  structure  to  prepare  for  the 
introduction  of  AR  systems.  Augmented  reality  is  not  just  another  smart-device  on  the 
battlefield.  It  is  a  means  to  transform  how  the  military  gathers  information,  analyzes  its  operating 
environment,  makes  decisions,  and  collaborates  across  the  farthest  edges  of  an  operating 
environment.  The  ultimate  ability  of  AR  to  bring  about  the  increased  combat  effectiveness  it 
promises  will  result  from  a  new  generation  of  Soldiers,  Sailors,  Marines,  and  Airmen  embracing 
its  potential  and  tailoring  its  capabilities  to  their  needs.  Their  training,  their  resources,  and  their 
attitudes  will  be  decisive  in  determining  AR’s  future.  Whole  new  ways  of  conducting  daily 
military  activities,  and  conducting  large-scale  combat  operations,  will  have  to  be  developed 
throughout  all  echelons,  if  the  military  wants  to  make  full  use  of  the  range  of  possibilities  future 
AR  systems  will  present. 


80 


Conclusion 


Augmented  reality  technologies  have  matured  to  a  point  that  they  are  ready  to  transition 
to  large-scale  commercial  adoption.  Over  the  next  five  to  ten  years,  the  use  of  AR  systems  will 
fundamentally  change  how  users  interact  with  each  other  and  their  environment.  Access  to  AR 
systems  will  alter  how  AR  users  perceive  their  surroundings  and  learn  new  information.  The 
growth  of  AR  systems  in  the  consumer  market  will  increase  calls  for  its  adoption  within  the 
ranks  of  the  military.  Many  will  discover  that  many  civilian  AR  uses-cases  overlap  with  military 
requirements.  As  such,  now  is  the  time  to  begin  considering,  and  preparing  for  the  use  of  AR 
systems  within  the  ranks  of  all  branches  of  service. 

For  combat  forces  at  the  tactical  level,  access  to  AR  information  will  provide  a  decided 
advantage  in  their  ability  to  observe,  fix,  and  finish  targets.  At  the  same  time,  AR  will  provide 
combat  forces  better  situational  awareness  and  access  to  control  measures  to  help  limit  fratricide 
and  collateral  damage.  The  use  of  AR  systems  also  has  the  potential  to  improve  decision-making. 
For  operational  leaders,  AR  can  provide  new  means  to  understand,  visual,  and  describe  complex 
problems  and  environments.  It  can  also  help  operational  leaders  direct  their  forces  and  assess  the 
effectiveness  of  their  actions.  Adopting  AR  systems  at  both  the  tactical  and  operational  levels 
will  have  its  challenges.  However,  failing  to  integrate  this  revolutionary  technology  will  cost  the 
U.S  military  its  decision-making  advantage  and  jeopardize  its  ability  to  achieve  information 
superiority  over  its  adversaries. 

The  U.S.  military  must  adopt  augmented  reality  (AR)  technologies  at  the  tactical  and 
operational  levels  or  risk  losing  a  decision-making  advantage  to  its  adversaries.  Augmented 
reality  systems  represent  a  revolution  in  information  technologies.  As  such,  they  offer  the  next 
iterative  advance  in  Network  Centric  Warfare  capabilities  that  the  U.S.  military  should  exploit. 
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Augmented  reality  systems  can  ensure  U.S.  forces  maintain  their  information  superiority 
advantage  throughout  this  next  revolution  in  information  technology. 
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